Generate QR code
Open Access
Subscription Access
Differential entropy and correlation analysis: a system approach
https://doi.org/10.26896/1028-6861-2026-92-1-77-86
Abstract
Every year we see an increase in the requirements for the efficiency of both newly created and existing systems. This leads to a complication of the tasks to be solved when creating, diagnosing, and managing these systems and phenomena. One of these tasks is to determine the nature and structure of relationships between ongoing processes. These processes are stochastic in nature, and they are usually described by correlation dependencies. However, their multidimensionality and multiple connectivity make it difficult to apply correlation analysis. For a clearer understanding and use of correlation dependencies, their systematic interpretation is necessary. A fundamental characteristic of any system with ambiguous or probabilistic behavior is entropy. Specifically, several results have been obtained linking the differential entropy of random vectors with correlation characteristics. The purpose of the article is to investigate and systematize the relationship between differential entropy and correlation indicators used in multidimensional statistical analysis. The article considers all the main variants of correlations in multidimensional systems, including the cross correlation between all elements, between subsystems, at different levels of the system, between one element (subsystem) and a group of elements (subsystems). In all cases, it has been established that correlation indices are analytically determined from the differential entropy of a system considered in the form of a random vector. Several examples have been given to illustrate the results obtained.
About the Author
A. N. Tyrsin
The first President of Russia B. N. Yeltsin Ural Federal University; Science and Engineering Center «Reliability and Resource of Large Systems and Machines», RAS, Ural Branch
Россия
Россия
Alexander N. Tyrsin
19, ul. Mira, Yekaterinburg, 620062
54a, Studencheskaya ul., Yekaterinburg, 620049
References
1. Loskutov A. Yu., Mikhailov A. S. Fundamentals of the theory of complex systems. 2nd edition. — Moscow – Izhevsk: Regulyarnaya i khaoticheskaya dinamika, 2019. — 619 p. [in Russian].
2. Smorodinskaya N. V. The complexity of the organization of economic systems in the context of nonlinear development / Vestn. Inst. Ékon. RAN. 2017. No. 5. P. 104 – 115 [in Russian].
3. Kalyaev I. A. Artificial intelligence: coming soon? / Ékon. Strat. 2019. Vol. 21. No. 5(163). P. 6 – 15 [in Russian]. DOI: 10.33917/es-5.163.2019.6-15
4. Stankovic M., Garba A. A., Neftenov N. Emerging technology trends: artificial intelligence and big data for development 4.0. — Geneva: International Telecommunication Union, 2021. — 108 p. https://www.itu.int/dms_pub/itu-d/opb/tnd/ D-TND-02-2021-PDF-R.pdf
5. Ayvazyan S. A., Yenyukov I. S., Meshalkin L. D. Applied statistics: a study of dependencies. — Moscow: Finansy i statistika, 1985. — 488 p. [in Russian].
6. Orlov A. I. Diversity of the models for regression analysis (generalizing article) / Industr. Lab. Mater. Diagn. 2018. Vol. 84. No. 5. P. 63 – 73 [in Russian]. DOI: 10.26896/1028-6861-2018-84-5-63-73
7. Adachi K. Matrix-based introduction to multivariate data analysis. 2nd edition. — Springer, 2020. — 457 p. DOI: 10.1007/978-981-15-4103-2
8. Härdle W. K., Simar L., Fengler M. R. Applied multivariate statistical analysis. 6th edition. — Springer, 2024. — 613 p. DOI: 10.1007/978-3-031-63833-6
9. Prangishvili I. V. Entropy and other systemic patterns: issues of management of complex systems. — Moscow: Nauka, 2003. — 428 p. [in Russian].
10. Matorin S. I., Zhikharev A. G. System-wide patterns as meaningful elements of a system theory based on a system-object approach / Nauch. Vedom. Belgorod. Gos. Univ. Ser. Ékon. Inform. 2018. Vol. 45. No. 2. P. 372 – 384 [in Russian]. DOI: 10.18413/2411-3808-2018-45-2-372-384
11. Mobus G. E. Systems science theory, analysis, modeling, and design. — Switzerland: Springer Nature, 2022. — 814 p. DOI: 10.1007/978-3-030-93482-8
12. Cox L. A. Risk analysis of complex and uncertain systems. — Springer, 2009. — 442 p. DOI: 10.1007/978-0-387-89014-2
13. Masaev S. N., Dorrer M. G. Methodology for assessing the company’s management system based on adaptive correlation to the external environment / Vestn. Sibir. Gos. Aérokosm. Univ. im. Akad. M. F. Reshetneva. 2009. Nos. 1 – 2(22). P. 157 – 160 [in Russian].
14. Gorban A. N., Tyukina T. A., Pokidysheva L. I., Smirnova E. V. Dynamic and thermodynamic models of adaptation / Phys. Life Rev. 2021. Vol. 37. P. 17 – 64. DOI: 10.1016/j.plrev.2021.03.001
15. Shpitonkov M. I. Application of correlation adaptometry in biomedical problems / Model. Dekompoz. Optim. Slozh. Din. Prots. 2022. Vol. 37. No. 1(37). P. 82 – 92 [in Russian]. DOI: 10.14357/24098639220106
16. Kurkin S. A., Smirnov N. M., Paunova R., et al. Beyond pairwise interactions: higher-order Q-analysis of fMRI-based brain functional networks in patients with major depressive disorder / IEEE Access. 2024. Vol. 12. P. 197168 – 197186. DOI: 10.1109/access.2024.3521249
17. Voigt J. Stochastic operators, information, and entropy / Comm. Math. Phys. 1981. Vol. 81. P. 31 – 38. DOI: 10.1007/bf01941799
18. Klimontovich Yu. L. Entropy and information of open systems / Physics-Uspekhi. 1999. Vol. 42. No. 4. P. 375 – 384. DOI: 10.1070/pu1999v042n04abeh000568
19. Skorobogatov S. M. Disasters and survivability of reinforced concrete structures (classification and elements of theory). — Yekaterinburg: UrGUPS, 2009. — 512 p. [in Russian].
20. Timashev S. A., Tyrsin A. N. Construction of a linear regression model based on the entropy approach / Industr. Lab. Mater. Diagn. 2009. Vol. 75. No. 3. P. 66 – 69 [in Russian].
21. Wilson A. G. Entropy in urban and regional modelling: retrospect and prospect / Geogr. Anal. 2010. Vol. 42. No. 4. P. 364 – 394. DOI: 10.1111/j.1538-4632.2010.00799.x
22. Jakimowicz A. The role of entropy in the development of economics / Entropy. 2020. Vol. 22. No. 4. 452. DOI: 10.3390/e22040452
23. Razmjooy N., Estrela V. V., Loschi H. J. Entropy-based breast cancer detection in digital mammograms using world cup optimization algorithm / Int. J. Swarm Intell. Res. 2020. Vol. 11. No. 3. P. 1 – 18. DOI: 10.4018/ijsir.2020070101
24. Zachary D., Dobson S. Urban development and complexity: Shannon entropy as a measure of diversity / Plan. Pract. Res. 2020. Vol. 36. No. 2. P. 157 – 173. DOI: 10.1080/02697459.2020.1852664
25. Shannon C. E. A mathematical theory of communication / The Bell Syst. Tech. J. 1948. Vol. 27. P. 379 – 423, 623 – 656.
26. Wentzel E. S. Theory of probability. 10th edition. — Moscow: Vysshaya shkola, 2006. — 575 p. [in Russian].
27. Gelfand I. M., Kolmogorov A. N., Yaglom A. M. Quantity of information and entropy for continuous distributions / Proc. of the III All-Union Math. Congr. Vol. 3. — Moscow: AN SSSR, 1958. P. 300 – 320 [in Russian].
28. Joe H. Relative entropy measures of multivariate dependence / J. Am. Stat. Assoc. 1989. Vol. 84. No. 405. P. 157 – 164. DOI: 10.1080/01621459.1989.10478751
29. Peña D., Van der Linde A. Dimensionless measures of variability and dependence for multivariate continuous distributions / Comm. Stat. Theory Meth. 2007. Vol. 36. No. 10. P. 1845 – 1854. DOI: 10.1080/03610920601126449
30. Peña D., Rodriguez J. Descriptive measures of multivariate scatter and linear dependence / J. Multivar. Anal. 2003. Vol. 58. P. 361 – 374. DOI: 10.1016/s0047-259x(02)00061-1
31. Tyrsin A. N. Measure of joint correlation dependence of multidimensional random variables / Industr. Lab. Mater. Diagn. 2014. Vol. 80. No. 1. P. 76 – 80.
32. Tyrsin A. N. Entropy modeling of multidimensional stochastic systems. — Voronezh: Nauchnaya kniga, 2016. — 156 p. [in Russian]. https://elibrary.ru/item.asp?id=25475510
33. Härdle W. Applied nonparametric regression. — Cambridge Univ. Press, 1992. — 434 p.
34. Tyrsin A. N. Scalar measure of the interdependence between random vectors / Industr. Lab. Mater. Diagn. 2018. Vol. 84. No. 7. P. 76 – 82 [in Russian]. DOI: 10.26896/1028-6861-2018-84-7-76-82
35. Tyrsin A. N. Scalar measure of the relationship between several random vectors / Industr. Lab. Mater. Diagn. 2022. Vol. 88. No. 3. P. 73 – 80 [in Russian]. DOI: 10.26896/1028-6861-2022-88-3-73-80
36. Tyrsin A. N., Shalkevich L. V., Ostroushko D. V., et al. Investigation of perinatal damage to the central nervous system in children in the neonatal period by methods of multidimensional statistical analysis / Sist. Anal. Upravl. Biomed. Sist. 2017. Vol. 16. No. 3. P. 595 – 605 [in Russian].
Review
For citations:
Tyrsin A.N. Differential entropy and correlation analysis: a system approach. Industrial laboratory. Diagnostics of materials. 2026;92(1):77-86. (In Russ.) https://doi.org/10.26896/1028-6861-2026-92-1-77-86
Views:
19
JATS XML
Email this article 