The constant growth of data volumes requires the development of effective methods for managing, processing, and storing information. Additionally, it is advisable to apply multimodal approaches for knowledge aggregation to extract additional knowledge. Usually, the problem of efficient processing of multimodal data is associated with high-quality data preprocessing. One of the most critical preprocessing steps is synchronizing multimodal data streams to analyze complex interactions in different data types. In this article, we evaluate existing approaches to synchronization, focusing on strategies based on real-time classifiers, which are based on comprehensive platforms for data integration and management. After the synchronization of multimodal sets, the key stage is data fusion, data identification in different channels, such as text, video, and audio. The results demonstrate the feasibility of the proposed synchronization approach for revealing subtle relationships between various data sets. An architectural solution was also suggested to integrate the proposed method into existing multimodal data processing pipelines. This work contributes to developing synchronization tools for multimodal data analysis in dynamic real world scenarios.
- Jun, S. Technology Integration and Analysis Using Boosting and Ensemble. J. Open Innov. Technol. Mark. Complex. 2021, 7, 27. https://doi.org/10.3390/joitmc7010027
- Chen, Z., Feng X., Zhang S. Emotion detection and face recognition of drivers in autonomous vehicles in IoT platform, Image and Vision Computing, Volume 128, 2022, https://doi.org/10.1016/j.imavis.2022.104569.
- Yih-Shiuan L., Wang C.. 2024. "A Cyber-Physical Testbed for IoT Microgrid Design and Validation" Electronics 13, no. 7: 1181. https://doi.org/10.3390/electronics13071181
- Havryliuk, M., Kaminskyy, R., Yemets, K., Lisovych, T. (2023). Interactive Information System for Automated Identification of Operator Personnel by Schulte Tables Based on Individual Time Series. In: Hu, Z., Zhang, Q., He, M. (eds) Advances in Artificial Systems for Logistics Engineering, Vol. 180. Springer, Cham, DOI: 10.1007/978-3-031-36115-9_34
- Basystiuk, O., Melnykova, N. and Rybchak, Z., 2023, June. Machine Learning Methods and Tools for Facial Recognition Based on Multimodal Approach. In MoMLeT+ DS (pp. 161-170).
- Strubytskyi R., Shakhovska N., Method and models for sentiment analysis and hidden propaganda finding, Computers in Human Behavior Reports, Volume 12, https://doi.org/10.1016/j.chbr.2023.100328.
- Dai, Z., Zakka, V.G., Manso, L.J.; Rudorfer, M.; Bernardet, U.; Zumer, J.; Kavakli-Thorne, M. Sensors, Techniques, and Future Trends of Human-Engagement-Enabled Applications: A Review. Algorithms 2024, 17, 560. https://doi.org/10.3390/a17120560
- Chen H., Ma H., Chu X., Xue D., Anomaly detection and critical attributes identification for products with multiple operating conditions based on isolation forest, Advanced Engineering Informatics, Volume 46, https://doi.org/10.1016/j.aei.2020.101139.
- Havryliuk, M., Hovdysh, N., Tolstyak, Y., Chopyak, V., & Kustra, N. (2023, November). Investigation of PNN Optimization Methods to Improve Classification Performance in Transplantation Medicine. In IDDM (pp. 338-345).
- Basystiuk O., Melnykova N., Rybchak Z. “Multimodal Learning Analytics: An Overview of the Data Collection Methodology,” IEEE 18th International Conference on Computer Science and Information Technologies, Lviv, Ukraine, 2023, pp. 1-4, DOI: 10.1109/CSIT61576.2023.10324177.
- Loaiza-Arias, M.; Álvarez-Meza, A.M.; Cárdenas-Peña, D.; Orozco-Gutierrez, Á.Á.; Castellanos-Dominguez, G. Multimodal Explainability Using Class Activation Maps and Canonical Correlation for MI-EEG Deep Learning Classification. Appl. Sci. 2024, 14, 11208. https://doi.org/10.3390/app142311208
- Su, Q.; Yao, Y.; Chen, C.; Chen, B. Generating a 30 m Hourly Land Surface Temperatures Based on Spatial Fusion Model and Machine Learning Algorithm. Sensors 2024, 24, 7424. https://doi.org/10.3390/s2423742
- Yakovyna V., Shakhovska N. "Software failure time series prediction with RBF, GRNN, and LSTM neural networks", Procedia Computer Science 207(4):837-847, DOI:10.1016/j.procs.2022.09.139.
- Paterega, I., Melnykova, N. (2024). Imbalanced data: a comparative analysis of classification enhancements using augmented data. European Science, 3(sge28-03), 54–72. https://doi.org/10.30890/2709-2313.2024-28-00-017.
- Basystiuk O., Melnykova N., Rybchak Z. "Multimodal Learning Analytics: An Overview of the Data Collection Methodology," 2023 IEEE 18th International Conference on Computer Science and Information Technologies (CSIT), Lviv, Ukraine, 2023, pp. 1-4, doi: 10.1109/CSIT61576.2023.10324177.
- Merino-Monge, M., Molina-Cantero, A. J., et al., (2020). An easy-to-use multi-source recording and synchronization software for experimental trials. IEEE Access, 8, 200618-200634.
- Govindarajan, Y., Ganesan, V. P. A., & Ramesh, D. (2024). Multi-modal biometric authentication: Leveraging shared layer architectures for enhanced security. arXiv preprint arXiv:2411.02112..
- Muhammad, T. (2022). A Comprehensive Study on Software-Defined Load Balancers: Architectural Flexibility & Application Service Delivery in On-Premises Ecosystems. International Journal of Computer Science and Technology, 6(1), 1-24.
- Zhaoyang N., Zhong G., Yu H. "A review on the attention mechanism of deep learning," Neurocomputing 452 (2021): 48-62.
- Basystiuk O., Melnykova N., Rybchak Z. "Detecting Multimodal Data in Information System," CSIT-2024: Computer Science and Information Technologies, 16-19 October 2024, Lviv, Ukraine.