An algorithm for constructing a multimodal identification model with cyclic learning for dynamic process with changing dynamics

This paper proposes a new method for identification of technological processes with changing dynamics. This approach involves statistical analysis of dependencies between the inputs and outputs of the process. An algorithm for process modes identification is based on probabilistic latent analysis and EM algorithm. A process with changing dynamics is described using a multimodal model, which consists of a mode classifier and a block of basic mode models. For each mode, an inductive knowledge base is formed, a binary mask of mode inputs is calculated using feature selection, and a basic mode model is constructed. The mode classifier is trained to determine the boundaries of each mode. To improve forecast accuracy, this paper proposes using cyclical training of the mode classifier and the block of basic mode models. The idea of cyclic learning is to iteratively train both parts of the multimodal model, where the prediction result of one part of the multimodal model is used to train the other. The classifier learns to most accurately determine the boundaries of process modes, and each basic model learns to most accurately describe its corresponding mode. Mode is determined by mode classifier for the current state and the corresponding basic mode model is used to predict output parameters. The proposed approach was compared with popular machine learning methods using real data from a multimodal iron ore flotation process. Based on numerical simulation results using test process data, the multimodal identification model demonstrated the highest prediction accuracy.

1. Gur’eva E. M., Kol’tsov A. G. The use of soft sensors to determine the quality of petroleum products / Din. Sist. Mekh. Mashin. 2016. No. 1. P. 296 – 301 [in Russian].

2. Torgashov A., Skogestad S. The use of first principles model for evaluation of adaptive soft sensor for multicomponent distillation unit / Chem. Eng. Res. Design. 2019. DOI: 10.1016/j.cherd.2019.08.017

3. Bikmukhametov T., Jaschke J. First principles and machine learning virtual flow metering: a literature review / J. Petrol. Sci. Eng. 2020. Vol. 184. P. 106487. DOI: 10.1016/j.petrol.2019.106487

4. Bakhtadze N. N. Virtual analyzers: identification approach / Autom. Remote Control. 2004. P. 1691 – 1709 [in Russian]. DOI: 10.1023/b:aurc.0000047885.52816.c7

5. Moir T. J. FIR System identification using feedback / J. Signal Inf. Proc. 2013. Vol. 4. No. 4. DOI: 10.4236/jsip.2013.44049

6. Nikolaou M., Vuthandam P. FIR model identification: parsimony through kernel compression with wavelets / AIChE J. 1998. Vol. 44(1). P. 141 – 150. DOI: 10.1002/aic.690440115

7. Ho B. L., Kalman R. E. Effective construction of linear state-variable models from input-output functions / Regelungstechnik. 1965. Vol. 12. P. 545 – 548. DOI: 10.1524/auto.1966.14.112.545

8. Aström K. J., Bohlin T. Numerical identification of linear dynamic systems from normal operating records / Proc. IFAC Symp. Self-Adapt. Syst. 1965. P. 96 – 111. DOI: 10.1016/s1474-6670(17)69024-4

9. Akaike H. Stochastic theory of minimal realization / IEEE Trans. Automat. Control. 1974. Vol. 26. P. 667 – 673. DOI: 10.1109/tac.1974.1100707

10. Box G. E. P., Jenkins G. M. Time Series Analysis, Forecasting and Control. — Oakland, CA: Holden-Day, 1976. — 553 p.

11. Aström K. J., Eykhoff P. System identification: a survey / Automatica. 1971. Vol. 7. P. 123 – 162. DOI: 10.1016/0005-1098(71)90059-8

12. Ljung L. On consistency and identifiability / Math Program Study. 1976. Vol. 5. P. 169 – 190. DOI: 10.1007/bfb0120772

13. Ljung L. Asymptotic variance expressions for identified black-box transfer function models / IEEE Trans. Automat. Control. 1985. Vol. AC-30. P. 834 – 844. DOI: 10.1109/cdc.1984.272156

14. Ljung L., Caines P. E. Asymptotic normality of prediction error estimators for approximative system models / Stochastics. 1979. Vol. 3. P. 29 – 46. DOI: 10.1109/cdc.1978.268066

15. Ljung L., Söderström T. Theory and Practice of Recursive Identification. — MIT Press, 1983. — 529 p.

16. Haykin S. Adaptive Filter Theory. 4th edition. — Prentice Hall, 2002. — 920 p.

17. Cao L., Wang J., Su J., et al. Comprehensive analysis on machine learning approaches for interpretable and stable soft sensors / IEEE Trans. Instr. Meas. 2025. P. 1 – 1. DOI: 10.1109/tim.2025.3556830

18. Shobana R., Kumar D. R., Jaint B. A novel locally connected recurrent neural network for identification of nonlinear dynamical system / 10th Int. Conf. on Signal Processing and Integrated Networks (SPIN). 2023. P. 430 – 435. DOI: 10.1109/icpeices62430.2024.10719117

19. Kaur J., Mahajan H. K., Singh S., Banerjee S. Recurrent neural network for aircraft parameter estimation / 6th Int. Conf. on Computing Communication and Automation (ICCCA). 2021. P. 604 – 608. DOI: 10.1109/iccca52192.2021.9666275

20. Hochreiter S., Schmidhuber J. Long short-term memory / Neural Comput. 1997. Vol. 9. No. 8. P. 1735 – 1780. DOI: 10.1162/neco.1997.9.8.1735

21. Cho K., Merrienboer B., Gulcehre C., et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation / Conf. on Empirical Methods in Natural Language Processing (EMNLP). 2014. P. 1724 – 1734. arXiv: 1406. 1078. DOI: 10.3115/v1/d14-1179

22. Vaswani A., Shazeer N., Parmar N., et al. Attention is all you need / Adv. Neural Inf. Proc. Syst. 2017. Vol. 30. arXiv: 1706. 03762. DOI: 10.48550/arXiv.1706.03762

23. Bachtadze N., Lototsky V., Maximov E., Pavlov B. Associative search models in industrial systems / IFAC Proc. Vol. 2007. Vol. 40(3). P. 105 – 108. DOI: 10.3182/20070523-3-es-4908.00018

24. Cybenko G. Just-in-time learning and estimation / Identification, Adaptation, Learning: The Science of Learning Models from Data. — Berlin: Springer, 1996. P. 423 – 434. DOI: 10.1007/978-3-662-03295-4_11

25. Kushnarev V. N. Increasing computational performance of virtual analyzers based on associative search using cluster models / Proc. of the XX All-Russian School-Conf. of Young Scientists «Management of Large-Scale Systems», September 10 – 13, 2024, Novocherkassk [in Russian].

26. Bakhtadze N., Lototsky V., Vlasov S., Sakrutina E. Associative search and wavelet analysis techniques in system identification / IFAC-PapersOnLine. 2012. Vol. 16. Part 1. P. 1227 – 1232. DOI: 10.3182/20120711-3-be-2027.00221

27. Jacobs R., Jordan M., Nowlan S., Hinton G. Adaptive mixtures of local experts / Neural Comput. 1991. Vol. 3. P. 79 – 87. DOI: 10.1162/neco.1991.3.1.79

28. Hofmann T. Probabilistic latent semantic indexing / Proc. of the 22nd Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval. — New York: ACM, 1999. P. 50 – 57. DOI: 10.1145/312624.312649

29. Vorontsov K. V., Potapenko A. A. Modifications of the EM algorithm for probabilistic topic modeling / Mach. Learn. Data Anal. 2013. Vol. 1. No. 6. P. 657 – 686 [in Russian].

30. Rokach L. Pattern Classification Using Ensemble Methods. — World Scientific, 2010. — 225 p. ISBN: 978-981-4271-06-6

31. Dempster A. P., Laird N. M., Rubin D. B. Maximum likelihood from incomplete data via the EM algorithm / J. Roy. Stat. Soc. Ser. B. 1977. Vol. 39(1). P. 1 – 38. DOI: 10.1111/j.2517-6161.1977.tb01600.x

32. John G. H., Kohavi R., Pfleger K. Wrappers for feature subset selection / Artif. Intelligence. 1997. Vol. 97. P. 273 – 324. DOI: 10.1016/s0004-3702(97)00043-x

33. Dash M., Choi K., Scheuermann P., Liu H. Feature selection for clustering: a filter solution / Second Int. Conf. on Data Mining. 2002. P. 115 – 122. DOI: 10.1109/icdm.2002.1183893

34. Tibshirani R. Regression shrinkage and selection via the lasso / J. Roy. Stat. Soc. Ser. B. 1996. Vol. 58(1). P. 267 – 288. DOI: 10.1111/j.2517-6161.1996.tb02080.x

35. Narendra P. M., Fukunaga K. A branch and bound algorithm for feature subset selection / IEEE Trans. Comput. 1977. Vol. 26(9). P. 917 – 922. DOI: 10.1109/tc.1977.1674939

36. Babatunde O., Armstrong L., Leng J., Diepeveen D. A Genetic algorithm-based feature selection / Int. J. Electron. Comm. Comput. Eng. 2014. Vol. 5. P. 889 – 905.

37. Goodfellow I., Pouget-Abadie J., Mirza M., et al. Generative adversarial networks / Adv. Neural Inf. Proc. Syst. 2014. Vol. 3. DOI: 10.1145/3422622

38. Fabisch A. GMR: Gaussian mixture regression / J. Open Source Software. 2021. Vol. 6. P. 3054. DOI: 10.21105/joss.03054