This article analyzes the use of microexpressions – subtle facial movements that are difficult for the human eye to notice, and even more difficult to immediately analyze, even specialists in the field do not always succeed in this perfectly, because their speed is only 1/5 to 1/3 of a second, for assessment of psychological state using artificial intelligence methods. The research is aimed at improving the analysis of micro-mimicry for accurate identification of emotions and psychological state. An overview of implemented technological solutions based on CNN was conducted, and a method for their improvement was found. An experimental test conducted on video recordings of people experiencing various emotions showed the high accuracy of the developed method in recognizing emotions and psychological state. Despite the challenges of the scarcity of microexpression datasets and the subtlety of facial movements, the paper presents a CapsuleNet model for microexpression recognition, builds a system architecture, and conducts testing. By combining three main data sets (SMIC, CASME II and SAMM) into a unified cross-database, the method developed in the work tests the possibility of generalization of the model by different subject characteristics. The performance of CapsuleNet, evaluated by cross-baseline benchmarking and Leave-One-Object-Out validation, significantly outperforms the baseline (LBP-TOP) and other improved of an CNN models. The paper shows that the performance of the developed model, determined by unweighted average recall and F1 scores, outperforms both the LBP-TOP baseline and other state-of-the-art CNN models. In a comprehensive microexpression recognition system. First, we process the data to identify the peak frames in the sequences and isolate the face region in these frames. These processed face images are then moved to CapsuleNet for the classification. The results of the work is to develop and complement methods of emotional artificial intelligence, offering new insights into micromimic assessment of psychological states that affect mental health, human-computer interaction, and social robotics. This technology has potential for development and expansion. This is an additional opportunity for companies that work with people and it is important for them to monitor their productivity, as it is directly related to the psychological state.
1. Matsumoto, D., Hwang, H. S. (2011). Evidence for training the ability to read microexpressions of emotion. Motivation Emotion, 35, 181–191. https://doi.org/10.1007/s11031-011-9212-2
2. Russell, T. A., Chu, E., & Phillips, M. L. (2006). A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. British Journal of Clinical Psychology, 45(4), 579583. https://doi.org/10.1348/014466505X90866
3. Weinberger, S. (2010). Airport security: intent deceive?. Nature, 465, 412–415. https://doi.org/10.1038/465412a
4. Ekman, Paul. (2009). Lie catching and microexpressions. In The Philosophy of Deception, ed C.W. Martin (pp. 118–133). Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195327939.003.0008
5. Ekman, P. "Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage," revised ed.; WW Norton Company: New York, NY, USA, 2009.
6. Davison, A. K., Lansley, C., Costen, N., Tan, K. & Yap, M. H. (2018). Samm: A spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing, 9(1), 116–129. https://doi.org/10.1109/TAFFC.2016.2573832
7. Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014). Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS one, 9(1). https://doi.org/10.1371/journal.pone.0086041
8. Li X, Pfister T, Huang X, Zhao G, Pietika inen M (2013). A Spontaneous Micro-expression Database: Inducement, Collection and Baseline, 10th Proc Int Conf Autom Face Gesture Recognit (FG2013). Shanghai, China. https://doi.org/10.1109/FG.2013.6553717
9. Davison, A., Merghani, W., & Yap, M. H. (2018). Objective classes for microfacial expression recognition. Journal of Imaging, 4(10), 119. https://doi.org/10.3390/jimaging4100119
10. Xi, E., Bing, S., & Jin, Y. (2017). Capsule network performance on complex data. arXiv preprint arXiv:1712.03480. https://doi.org/10.48550/arXiv.1712.03480
11. Zhao, G., & Pietikainen, M. (2007). Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. on PAMI, 29(6), 915–928. https://doi.org/10.1109/TPAMI.2007.1110
12. Wang, Y., See, J., Phan, C.W. et al. (2014). LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Microexpression Recognition, 12th Asian Conference on Computer Vision 2014, Singapore, Singapore. https://doi.org/10.1007/978-3-319-16865-4_34
13. Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., & Pietikäinen, M. (2017). Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods, IEEE Transactions on Affective Computing, 9(4), 563–577. https://doi.org/10.1109/TAFFC.2017.2667642
14. Afshar, P., Mohammadi, A., & Plataniotis, K. N. (2018). Brain tumor type classification via capsule networks. 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, pp. 3129–3133. https://doi.org/10.1109/ICIP.2018.8451379
15. Borza, Diana, Danescu, Radu, Itu, Razvan, & Darabant, Adrian Sergiu. (2017). High-Speed Video System for Micro-Expression Detection and Recognition. Sensors 2017, 17(12), 2913. https://doi.org/10.3390/s17122913
16. Sabour, S., Frosst, N., & Hinton, G. E. (2017). Dynamic Routing Between Capsules. ArXiv, abs/1710.09829. https://doi.org/10.48550/arXiv.1710.09829
17. Simonyan, K., & Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. ArXiv preprint arXiv:1512.03385. https://doi.org/10.48550/arXiv.1409.1556
18. Wang, Y., Sun, A., Han, J., Liu, Y., & Zhu, X. (2018). Sentiment analysis by capsules. WWW'18: Proceedings of the 2018 World Wide Web Conference, Lyon, France, 1165–1174. https://doi.org/10.1145/3178876.3186015