Statistics of words occurrences in natural and random texts

Authors:

Oleg Kushnir, Mykola Alfavitskyi, Viktor Dzikovskyi, Lyubomyr Ivanitskyi, Sergiy Rykhlyuk, Volodymyr Sokulskyi

Department of Optoelectronics and Information Technologies, Ivan Franko National University of Lviv Natural Sciences College of Ivan Franko National University of Lviv

We study experimentally statistical distributions that describe the appearance of words in a number of natural texts, as well as in the random texts derived on their basis. It is shown that the probability mass function of the respective intervals between words is practically the same for the natural and random texts and manifests a fat tail, which is inconsistent with purely stochastic character of those intervals. Significant deviations of the vocabulary growth dynamics found for the natural and random texts from the dynamics predicted by the power Heaps’ law, together with a crossover found in the dictionary of one of the natural texts, confirm a need in generalization of that law.

natural texts

random texts

statistical laws of linguistics

vocabulary

fat-tailed distributions

1. Shcherba L. V. Izbrannye raboty po iazykoznaniiu i fonetike, L. V. Shcherba, L : Izd-vo LHU, 1958, 182 p.

2. Rozental D. E., Telenkova M. A. Slovar trudnostei russkoho iazyka:o. 30 000 slov, 3-e izd., dop, M. : Rus. iaz., 1984, 704 p.

3. Vinohradov V. V. Osnovnye tipy leksicheskikh znachenii slova, Izbrannye trudy. Leksikolohiia i leksikohrafiia, M., 1977. –P. 162–189.

4. Vyskushenko S. A. Do problemy rozrobky dvomovnoho terminolohichnoho tematychnoho slovnyka anhliiskoi fakhovoi movy tvarynnytstva, S. A. Vyskushenko, Naukovyi visnyk Volynskoho natsionalnoho universytetu im. Lesi Ukrainky. Filolohichni nauky. Movoznavstvo : zb. nauk. prats, Lutsk : Volynskyi natsionalnyi un-t im. Lesi Ukrainky, 2011, No 3, Ch. 2, P. 194–197.

5. Kudashev I. S. Proektirovanie perevodcheskikh slovarei spetsialnoi leksiki : monohrafiia, I. S. Kudashev, Helsinki, 2007, 443 p.

6. Perebyinis V. Anhlo-ukrainskyi navchalnyi slovnyk z metodychnymy komentariamy ta hramatychnymy tablytsiamy, K. : Vezha, 2002, 424 p.

7. Radziievska S. Pryntsyp ukladannia anhlo-ukrainskoho slovnyka-dovidnyka z nanonauky, S. Radziievska, I. Chekman, Terminolohichnyi visnyk : zb. nauk. prats, K. : Instytut ukrainskoi movy NANU, 2013, Iss. 2 (1), P. 171–179.

8. Seliverstova K. T. Pidhotovka haluzevykh slovnykiv: zdobutky i osoblyvosti metodyky ukladannia perekladnoho slovnyka [Electronic resource], K. T. Seliverstova, Arkhivy Ukrainy : nauk.-prakt. zhurnal, 2013, No 5, P. 25–62, Access mode: http://nbuv.gov.ua/j-pdf/ay_2013_5_4.pdf.

9. Shyrokov V. A. Elementy leksykohrafii, K., Dovira, 2005, 304 p.

10. Shyrokov V. A. Informatsiina teoriia leksykohrafichnykh system, K., Dovira, 1998, 331 p.

11. Shyrokov V. A. Fenomenolohiia leksykohrafichnykh system, K., Nauk. dumka, 2004, 327 p.

12. Kucherenko O. Rozbudova suchasnoi natsionalnoi terminolohii tsyvilnoho zakhystu, O. Kucherenko, Visnyk Nats. un-tu "Lvivska politekhnika". Seriia "Problemy ukrainskoi terminolohii", 2009, No 648, P. 92–94.

13. Symonenko L. O. Biolohichna terminolohiia: formuvannia ta funktsionuvannia : tutorial, L. O. Symonenko; Uman. derzh. ped. un-t im. P. Tychyny, Uman : RVTs "Sofiia", 2006. –103 c.

14. Shunevych B. Suchasni sposoby vidboru terminiv ta ukladannia perekladnykh slovnykiv novykh terminosystem, B. I. Shunevych, Visnyk Zhytomyrskoho derzh. un-tu im. Ivana Franka, Iss. 38, P. 90–93.

15. Kulchytskyi I. M. Kompiuterno-tekhnolohichni aspekty stvorennia suchasnykh leksykohrafichnykh system, I. M. Kulchyts- kyi; Ukr. mov.-inform. fond NAN Ukrainy, Kyiv : Nats. b-ka Ukrainy im. V. I. Vernadskoho NAN Ukrainy, 2002, 59 c.

16. Kulchytskyi I. M. Vidobrazhennia strukturnykh vidnoshen reiestru terminolohichnykh slovnykiv u bazakh danykh, I. M. Kulchytskyi, Visn. Nats. un-tu "Lviv. politekhnika", 2010, No 689, P. 271–280.

17. Kulchytskyi I. M. Elektronnyi hlosarii z ar- khivistyky, I. M. Kulchytskyi, Visn. Nats. un-tu "Lviv. politekhnika", 2009, No 653, P. 148–156.

18. Dmytruk M. V. Formuvannia ukrainskoi veterynarnoi leksyky: avtoreferat dys. na zdobuttia nauk. stupenia kand. filol. nauk, M. V. Dmytruk, Zaporizhzhia: ZDU, 2001, 20 p.

19. Serbenska O. A. Rozvytok ukrainskoi yurydychnoi terminolohii pislia Velykoi Zhovtnevoi Sotsialistychnoi Revoliutsii: dys. … kand. filol. nauk: 10.02.01, O. A. Serbenska, Lviv, 1965. –465 p.

20. Antonovych Ye. A. Rosiisko-ukrainskyi slovnyk-dovidnyk z inzhenernoi hrafiky, dyzainu ta arkhitektury: tutorial, Ye. A. Antonovych, Ya. V. Vasylyshyn, V. A. Shpilchak, Lviv: Svit,200, 240 p.

21. Palahyn A. V., Yakovlev Yu. S. Systemnaia yntehratsyia sredstv kompiuternoi tekhnyky, Vynnytsa: UNIVERSUM, 2005, 680 p.

22. Morkovkin V. V. Ideohraficheskie slovari, V. V. Morkovkin, M. : Izd-vo Mosk. un-ta, 1970, 71 p.

23. Karaulov Iu. N. Linhvisticheskoe konstruirovanie i tezaurus literaturnoho iazyka, Iu. N. Karaulov, M. : Nauka, 1981, 363 p.

24. Balli Sh. Obshchaia linhvistika i voprosy frantsuzskoho iazyka, Sh. Balli, M. : Izd-vo inostr. literatury, 1955, 416 p.

25. Palahin A. V. Ontolohicheskie metody i sredstva obrabotki predmetnykh znanii: monohrafiia, A. V. Palahin, S. L. Kryvyi, N. H. Petrenko, Luhansk: izd-vo VNU im. V. Dalia, 2012, 324 p.

26. Potapova E. V. Linhvisticheskaia ontolohiia "Fizika mahnitnykh iavlenii" i ee prilozheniia: dis. ... kand. tekhn. nauk, K., 2013, 180 p.

27. Lande D. V. Elementy kompiuternoi linhvistyky v pravovii informatytsi, K., NDIIP NAPrN Ukrainy, 2014. –168 p.

28. Shyrokov V. A. Informatsiina teoriia leksykohrafichnykh system, K., Dovira, 1998, P. 80.

29. PAS 180 Smart city technology, Access mode: http://www.bsigroup.com/smart-cities/Smart- Cities-Standards-and-Publication/PAS-180-smart-cities-terminology/.

30. PAS 181 Smart city framework, Access mode: http://www.bsigroup.com/en-GB/smart-cities/Smart-Cities-Standards-and-Pu....

31. PAS 182:2014 Smart city concept model. Guide to establishing a model for data interoperability, Access mode: http://shop.bsigroup.com/Product Detail/?pid.

32. PAS 183 Smart cities – Guide to establishing a decision-making framework for sharing data and information services, Access mode: www.cibse.org/getmedia/96c2ff0e-73b6-4544-aebd-7debabe211d4/Draft-PAS-18....

33. PAS 8101:2014 Smart cities. Guide to the role of the planning and development process, Access mode: http://shop.bsigroup.com/ProductDetail/?pid=000000000030294642.

34. ISO 37120:2014 Sustainable development of communities -- Indicators for city services and quality of life( 2014), Access mode:https://www.iso.org/obp/ui/#iso:std:62436:en.

35. ISO/TR 37150:2014(en) Smart community infrastructures – Review of existing activities relevant to metrics, Access mode:https://www.iso.org/obp/ui/#iso:std:iso:tr:37150:ed-1:v1:en.

36. ISO/TS 37151:2015(en) Smart community infrastructures – Principles and requirements for performance etrics [Electronic resource], Access mode : https://www.iso.org/obp/ui/#iso:std:iso:ts:37151:ed-1:v1:en.

37. ISO/PRF TR 37152 Smart community infrastructures – Common framework for development and operation – Ad hoc group report [Electronic resource], Access mode : http://www.iso.org/iso/catalogue_detail.htm?csnumber=66898.

38. A collaborative approach to smart city transformation, Access mode: smartdubai.ae/whitepaper/Smart_Dubai_WhitePaper.pdf.

39. Al-Hader, M., & Rodzi, A. (2009). The smart city infrastructure development & monitoring. Theoretical and Empirical Researches in Urban Management, 4(2), 87–94.

40. Barrionuevo J. M., Berrone P., Ricart J. E. Smart Cities, Sustainable Progress /J. M. Barrionuevo, P. Berrone, J. E. Ricart, IESE Insight.-2012, Vol. 14, P. 50–57.

41. Batty, M., Axhausen, K. W., Giannotti, F., Pozdnoukhov, A., Bazzani, A., Wachowicz, M., Ouzounis, G. and Portugali, Y. (2012) Smart cities of the future. European Physical Journal Special Topics 214:481–518.

42. Benevolo C., Dameri R. P. and D’Auria B. Smart Mobility in Smart City, Access mode:file:///C:/Users/Admin/Downloads/9783319237831-P.2 %20(1).pdf.

43. Borja, J. (2007). Counterpoint: Intelligent cities and innovative cities. Universitat Oberta de Catalunya (UOC) Papers: E-Journal on the Knowledge Society, 5, Access mode: http://www.uoc.edu/uocpapers/5/dt/eng/mitchell.pdf.

44. Building a Smart + Equitable City, Access mode: http://www1.nyc.gov/assets/forward/documents/NYC-Smart-Equitable-City-Fi....

45. Caragliu A. Smart Cities in Europe, Andrea Caragliu, Chiara Del Bo, and Peter Nijkamp, Journal of Urban Technology, 2011, No 2, P. 65–82.

46. Caroline Colldahl, Sonya Frey, Joseph E. Kelemen. Smart Cities: Strategic Sustainable Development for an Urban World, School of Engineering Blekinge Institute of Technology Karlskrona, Sweden, 2013.

47. Eco-City Movement, Access mode: http://ecocity.ncr.vt.edu/ecocitymovement.html.

48. Deakin M. From Intelligent to Smart Cities, Mark Deakin, Husam Al Waer, Intelligent Buildings International, 2011, P. 140–152.

49. Getting Smart About Smart Cities, Access mode: http://sustainablecommunitiesleadershipacademy.org/resource_files/docume... %20Cities %20RG %20(2).pdf.

50. Giffinger R. Smart Cities Ranking: An Effective Instrument for the Positioning of Cities?, R. Giffinger, ACE Architecture City and Environment, 2010, Vol. 4, N 12, R. 7–25.

51. Global Innovators: International Case Studies on Smart Cities Smart Cities Study, October, 2013.

52. Gontar B., Gontar Z., Pamuła A. Deployment of Smart City Concept in Poland. Selected Aspects. Access mode:http://eltalpykla.vdu.lt:8080/bitstream/handle/1/963/ISSN2335-8750_2013_....

53. Green Economy Report, Access mode:http://web.unep.org/greeneconomy/resources/green-economy-report.

54. Green Technology – What is it? – Access mode: http://www.green-technology.org/what.htm.

55. Gruber T. R. A Translation Approach to Portable Ontology Specifications, Knowledge Acquisition, 1993, N 5, P. 199–220.

56. ICT4 Green by Donato Toppeta. How ICT can address Sustainability, Access mode: https://ict4green.wordpress.com/category/smart-city-2/.

57. Jadoul M. (2014) Smart cities are built on smart networks, Marc Jadoul, Access mode: https://insight.nokia.com/smart-cities-are-built-smartnetworks.

58. Jahan Selim Human Development Report. Work for Human Development, Selim Jahan, New York, 2015, 288 p.

59. Kearmey A. T. Global cities 2016, Mike Hales, Erik R. Peterson, Andrés Mendoza Peña, Nicole Dessibourg, NY, 2016, 13 p.

60. Komninos N. Smart Cities and the Future Internet in Europe, N. Komninos, Schaffers H., Journal of Knowledge Economy, 2013, No. 2. –P. 119–134.

61. Low energy, passive and zero-energy houses, Access mode: https://www.ourenergy.com/low_energy_passive_and_zero_energy_houses.html.

62. Making cities smarter Guide for city leaders: Summary of PD 8100 – Access mode: https://www.bsigroup.com/LocalFiles/en-GB/smartcities/resources/BSI-Maki....

63. Manville Catriona, Cochrane Gavin, Cave Jonathan Mapping Smart Cities in the EU/ Catriona Manville, Gavin Cochrane, Jonathan Cave, Brussels, 2014, 196 p.

64. Recommendations to the European Council: Europe and the Global Information Society, M. Bangemann and others, 24–25 June, Korfu, 1994.

65. Report on the WSIS Stocktaking 2016, ITU 201, 177 p.

66. Sekhar N. Kondepudi. Smart sustainable cities: An analysis of definitions, ITU, 2014.

67. Smart cities: Background paper. October 2013, Access mode:https://www.gov.uk/government/uploads/system/uploads/attachment_data/fil....

68. Smart Cities Maturity Model and Self-Assessment Tool, Access mode: www.scottishcities.org.uk/site/assets/files/1103/smart_cities_readiness_....

69. Smart cities. Ranking of European medium-sized cities, Access mode:http://www.smart-cities.eu/download/smart_cities_final_report.pdf.

70. Smart cities readiness guide. The planning manual for building tomorrow’s cities today, Access mode:www.swenergy.org/Data/Sites/1/media/documents/programs/government/SmartC....

71. Smart Cities: Technological Challenges and Issues Challenges and Issues, Access mode: http://www.cse.wustl.edu/~jain/talks/ftp/smrtcit.pdf.

72. Smart Cities. Transforming the 21st century city via the creative use of technology, Access mode:http://publications.arup.com/publications/s/smart_cities.

73. Smart city opportunities for London, Access mode: https://www.london.gov.uk/sites/default/files/arup-gla_smart_city_opport....

74. Smart City Wien. Framework Strategy, Vienna, 2014, 110 p.

75. "Smart+Connected City Services", Cisco Consulting Services, 2011, http:// www.cisco.com/web/about/ac79/docs/ps/Busan-Green-u-City_IBSG.pdf.

76. Smart Economy: Economía Inteligente, Access mode:https://web.ua.es/en/smart/smart-economy-economia-inteligente.html.

77. Smarter Cities. New cognitive approaches to long-standing challenges, Access mode: http://www.ibm.com/smarterplanet/ru/ru/smarter_cities/overview/.

78. Smart Sensor Networks: Technologies and Applications for Green Growth, Access mode: http://dx.doi.org/10.1787/5kml6x0m5vkh-en.

79. The Internet of Things: A Prototechnateby Enrique Lescure, Access mode: https://eoshorizon.wordpress.com/tag/smart-cities/.

80. The MK: Smart Project, Access mode: https:// mksmart.org.

81. The Role of Standards in Smart Cities, Access mode: https://www.bsigroup.com/LocalFiles/en-GB/smart-cities/resources/The-Rol....

82. The Smart Cities Mission, Access mode:www.slideshare.net/neersee/smart-city-presentation-50456375?next_slidesh....

83. The Smart City – City of Stockholm, Access mode: http://international.stockholm.se/city-development/the-smart-city/.

84. The Water Footprint Assessment Manual. Setting the Global Standard, London, 2011, 228 p.

85. Towards nearly zero-energy buildings, Access mode: https:/ec.europa.eu/energy/sites/ener/files/documents/nzeb_executive_summary.pdf.

86. Trends in Smart City Development, Access mode:www.nlc.org/Documents/Find %20City %20Solutions/Research %20Innovation/Smart %20Cities/Trends%20in %20Smart %20City %20Development.pdf.

87. United States Green Buildings Council, Access mode: http://www.usgbc.org/articles/what-green-building.

88. What is Green Infrastructure? – Access mode: https://www.epa.gov/green-infrastructure/what-green-infrastructure.

89. Bondarchuk A. P. Osnovy infokomunikatsiinykh tekhnolohii, A. P. Bondarchuk, H. S. Srochynska, M. H. Tverdokhlib, K.,2015, 76 p.

90. Hanin O. B. "Umnyi horod": perspektivy i tendentsii razvitiia, O. B. Hanin, Zhurnal ARS Administrandi, 2014, No 1, P. 124–135.

91. Ekolohiia zhyttia, Access mode: http://www.eco-live.com.ua/content/blogs/vodniy-slid-krain-svitu.

92. Eko misto, Access mode:http://ecotown.com.ua/.

93. Zinchenko A. Obzor initsiativ "smart city" 2014: indikatory effektivnosti, Aleksei Zinchenko, Access mode: http://www.researchclub.com.ua/journal/373.

94. Intelektualni transportni systemy, Access mode: http://www.sutp.org/files/contents/ documents/ resources/ A_Sourcebook/SB4_Vehicles-and-Fuels/GIZ_SUTP_SB4e_Intelligent-Transport-Systems_UA.pdf.

95. Kozlenko O. Tsyfrovi mista, K., 2008, 180 p.

96. Kupriianovskii V. P., Martynov B. A., Kornilev K. H. "Umnaia planeta" – kak eto sdelano? – Access mode:https://www.esri-cis.ru/news/arcreview/detail.php?ID= 16987&SECTION_ID=1048.

97. Pavlov O. A. Vid informatsiinoho do tsyfrovoho suspilstva: hlobalna stratehiia rozvytku smart-planety, Access mode: file:///C:/Users/Admin/Downloads/Pavlov_ %20Vid_ %20informatsiinoho_ %20do_ %20tsyfrovoho %20suspilstva %20(5).pdf.

98. Skalii I. Rozumne misto – misto maibutnoho, Access mode: msdp.undp.org.ua/data/publications/Skalii %20[Compatibility %20Mode].pdf.

99. Sokolovska O. O. Smart City: vykorystannia informatsiino- komunikatyvnykh tekhnolohii u mistsevomu samovriaduvanni [Electronic resource], O. O. Sokolovska, Aspekty publichnoho upravlinnia, 2014, No 11–12, P. 77–85, Access mode:http://nbuv.gov.ua/UJRN/aplup_2014_11-12_12.

100. Tekhnichne rehuliuvannia u sferi standartyzatsii "rozumnykh hromad" ta "rozumnykh mist": yevropeiskyi dosvid": Analitychna zapyska, Access mode: http://www.niss.gov.ua/articles/2338/.

101. Lendau Sydni I. Slovnyky: mystetstvo ta remeslo leksykohrafii, transl. from English Olhy Kocherhy, K., K. I. S., 2012, 480 p.

102. Iskusstvennyi intellekt: v 3 kn. Kn. 1: Sistemy obshcheniia i ekspertnye sistemy: [spravochnik, nauchn. red. E. V. Popov], M., Radio i sviaz, 1990, 464 p.

103. Hladun A. Ia. Ontolohii v korporativnykh sistemakh [Text], A. Ia. Hladun, Iu. V. Rohushina, Korporativnye sistemy, 2006, No 1, P. 41–47.