semantic analysis

The article addresses a currently important problem of natural language processing, the development of methods for assessing repetitiveness in textual documents and the empirical clarification of the resources of these methods for analyzing the presence of semantic load in texts. So far, the approaches based on the laws of statistical linguistics such as Zipf’s, Pareto’s and Heaps’ laws have been mainly used for this aim, as well as the analysis of word-clustering phenomena and long-term correlations of word tokens.

The article addresses the problem of text classification in the context of growing information flows and the need for automated content analysis. A universal information technology is proposed, combining classical machine learning methods with the potential of Large Language Models for processing news, scientific, literary, journalistic and legal texts. Using the BBC News corpus (2225 texts), k-means clustering with TF-IDF demonstrated clear thematic grouping.

In recent years, social media platforms have become powerful data collection tools to improve user experience. The vast amount of data generated through social media provides a unique opportunity to develop innovative recommendation systems. This article analyzes the application of data mining methods for social networks in the context of effective recommendation systems, focusing on three key methodologies: sentiment analysis (SA), topic modeling (TM) and social network analysis (SNA), highlighting their positive features.

The article is devoted to the use of semantic analysis in the generalization of text documents. The analysis of features of the most widespread methods of generalization of text documents and an estimation of quality of results of an estimation is carried out. Features of the improved method of annotative generalization of text documents, which uses the principles of hidden semantic analysis and elements of fuzzy logic to identify semantically important sentences, are presented.