DeepFaceEmotion: Enhancing Emotion Detection through Transfer Learning and Convolutional Neural Networks

Emerging as a crucial tool for companies in many industries, emotion recognition offers a greater understanding of consumer views of goods and services, hence strengthening client connections, improving service delivery, and guiding emotionally intelligent marketing strategies.  Particularly with the use of transfer learning techniques, end-to-end image-based emotion categorization has gained popularity in the last several years.  In tackling such challenging tasks, deep learning models have shown great efficacy.  Though reading emotions from visual data presents difficulties, the discipline still provides great room for creativity.  This paper investigates the classification of emotional expressions in pictures using sophisticated transfer learning models: EfficientNet, ResNet50V2, VGG-19, and DenseNet-121 as well as a traditional Convolutional Neural Network (CNN) architecture.  To find the most resilient technique, model performance is assessed using conventional measures like accuracy, precision, recall, and F1-score.  The findings show that the suggested approach achieves a classification accuracy of 95.7%.  These results imply that the method is widely relevant in several business sectors such as finance, marketing, education, healthcare, user experience, digital communication, and public mood monitoring.

emotion classification
глибоке навчання
transfer learning
детекція
  1. Kumar G. M. M. K., Indian A.  Categorizing Sentiment Polarities in Social Networks Data Using Convolutional Neural Network.  SN Computer Science.  3 (2), 116 (2021).
  2. Shoumy N. J., Ang L.-M., Seng K. P., Rahaman D. M. M., Zia T.  Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals.  Journal of Network and Computer Applications.  149, 102447 (2020).
  3. Gunter B., Koteyko N., Atanasova D.  Sentiment analysis: A market-relevant and reliable measure of public feeling?  International Journal of Market Research.  56 (2), 231–247 (2014).
  4. Sufi F. K.  Identifying the drivers of negative news with sentiment, entity and regression analysis.  International Journal of Information Management Data Insights.  2 (1), 100074 (2022).
  5. Hicham N., Nassera H.  Improving emotion classification in e-commerce customer review analysis using GPT and meta-ensemble deep learning technique for multilingual system.  Multimedia Tools and Applications.  83, 87323–87367 (2024).
  6. Alom M. Z., Taha T. M., Yakopcic C., Westberg S., Sidike P., Nasrin M. S., Hasan M., Van Essen B. C., Awwal A. A. S., Asari V. K.  A State-of-the-Art Survey on Deep Learning Theory and Architectures.  Electronics.  8 (3), 292 (2019).
  7. Ensafi Y., Amin S. H., Shah B.  Time-series forecasting of seasonal items sales using machine learning – A comparative analysis.  International Journal of Information Management Data Insights.  2 (1), 100058 (2022).
  8. Anand K., Urolagin S., Mishra R. K.  How does hand gestures in videos impact social media engagement – Insights based on deep learning.  International Journal of Information Management Data Insights.  1 (2), 100036 (2021).
  9. Dash M. K., Sahu M. K., Bhattacharyya J., Sakshi S.  Introduction: Customer-Centric Retailing in Emerging Markets.  Customer-Centricity in Organized Retailing.  3–8 (2023).
  10. Hicham N., Karim S., Habbat N.  Enhancing Arabic Sentiment Analysis in E-Commerce Reviews on Social Media Through a Stacked Ensemble Deep Learning Approach.  Mathematical Modelling of Engineering Problems.  10 (3), 790–798 (2023).
  11. Litjens G., Kooi T., Bejnordi B. E., Setio A. A. A., Ciompi F., Ghafoorian M., van der Laak Jeroen A. W. M., van Ginneken B., Sánchez C. I.  A survey on deep learning in medical image analysis.  Medical Image Analysis.  42, 60–88 (2017).
  12. Kaur H., Ahsaan S. U., Alankar B., Chang V.  A Proposed Sentiment Analysis Deep Learning Algorithm for Analyzing COVID-19 Tweets.  Information Systems Frontiers.  23 (6), 1417–1429 (2021).
  13. Hicham N., Nassera H., Karim S.  Enhancing Arabic E-Commerce Review Sentiment Analysis Using a hybrid Deep Learning Model and FastText word embedding.  EAI Endorsed Transactions on Internet of Things.  10, 1–10 (2023).
  14. Hicham N., Nassera H.  Towards Improving E-Commerce Customer Review Analysis for Arabic Language Opinion Mining.  International Journal of Computing.  23 (3), 387–395 (2024).
  15. Leon S. A., Altmann L. J. P., Abrams L., Gonzalez Rothi R. L. J., Heilman K. M.  Novel associative processing and aging: Effect on creative production.  Aging, Neuropsychology, and Cognition.  26 (6), 807–822 (2018).
  16. Zhao Z., Xu S., Kang B. H., Kabir M. M. J., Liu Y., Wasinger R.  Investigation and improvement of multi-layer perceptron neural networks for credit scoring.  Expert Systems with Applications.  42 (7), 3508–3516 (2015).
  17. Pranav E., Kamal S., Satheesh Chandran C., Supriya M. H.  Facial Emotion Recognition Using Deep Convolutional Neural Network.  2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS). 317–320 (2020).
  18. Wen G., Hou Z., Li H., Li D., Jiang L., Xun E.  Ensemble of Deep Neural Networks with Probability-Based Fusion for Facial Expression Recognition.  Cognitive Computation.  9 (5), 597–610 (2017).
  19. Jain N., Kumar S., Kumar A., Shamsolmoali P., Zareapoor M.  Hybrid deep neural networks for face emotion recognition.  Pattern Recognition Letters.  115, 101–106 (2018).
  20. Shaees S., Naeem H., Arslan M., Naeem M. R., Ali S. H., Aldabbas H.  Facial Emotion Recognition Using Transfer Learning.  2020 International Conference on Computing and Information Technology (ICCIT-1441).  1–5 (2020).
  21. Porcu S., Floris A., Atzori L.  Evaluation of Data Augmentation Techniques for Facial Expression Recognition Systems.  Electronics.  9 (11), 1892 (2020).
  22. Shi M., Xu L., Chen X.  A Novel Facial Expression Intelligent Recognition Method Using Improved Convolutional Neural Network.  IEEE Access.  8, 57606–57614 (2020).
  23. Hua W., Dai F., Huang L., Xiong J., Gui G.  HERO: Human Emotions Recognition for Realizing Intelligent Internet of Things.  IEEE Access.  7, 24321–24332 (2019).
  24. Zheng H., Wang R., Ji W., Zong M., Wong W. K., Lai Z., Lv H.  Discriminative deep multi-task learning for facial expression recognition.  Information Sciences.  533, 60–71 (2020).
  25. Boughida A., Kouahla M. N., Lafifi Y.  A novel approach for facial expression recognition based on Gabor filters and genetic algorithm.  Evolving Systems.  13 (2), 331–345 (2021).
  26. Haseena S., Saroja S., Madavan R., Karthick A., Pant B., Kifetew M.  Prediction of the Age and Gender Based on Human Face Images Based on Deep Learning Algorithm.  Computational and Mathematical Methods in Medicine.  2022, 1–16 (2022).
  27. Yaman D., Eyiokur F. I., Ekenel H. K.  Multimodal soft biometrics: combining ear and face biometrics for age and gender classification.  Multimedia Tools and Applications.  81 (16), 22695–22713 (2022).
  28. Duan M., Li K., Yang C., Li K.  A hybrid deep learning CNN-ELM for age and gender classification.  Neurocomputing.  275, 448–461 (2018).
  29. Waris F., Da F., Liu S.  Deep learning based features extraction for facial gender classification using ensemble of machine learning technique.  Multimedia Systems.  30 (4), 200 (2024).
  30. Gokulakrishnan S., Chakrabarti P., Hung B. T., Shankar S. S.  An optimized facial recognition model for identifying criminal activities using deep learning strategy.  International Journal of Information Technology.  15 (7), 3907–3921 (2023).
  31. Foggia P., Greco A., Saggese A., Vento M.  Multi-task learning on the edge for effective gender, age, ethnicity and emotion recognition.  Engineering Applications of Artificial Intelligence.  118, 105651 (2023).
  32. Gupta S.  Deep learning based human activity recognition (HAR) using wearable sensor data.  International Journal of Information Management Data Insights.  1 (2), 100046 (2021).
  33. Nasir J. A., Khan O. S., Varlamis I.  Fake news detection: A hybrid CNN-RNN based deep learning approach.  International Journal of Information Management Data Insights.  1 (1), 100007 (2021).
  34. Wen L., Li X., Li X., Gao L.  A New Transfer Learning Based on VGG-19 Network for Fault Diagnosis.  2019 IEEE 23rd International Conference on Computer Supported Cooperative Work in Design (CSCWD).  205–209 (2019).
  35. Theckedath D., Sedamkar R. R.  Detecting Affect States Using VGG16, ResNet50 and SE-ResNet50 Networks.  SN Computer Science.  1 (2), 79 (2020).
  36. Nandhini S., Ashokkumar K.  An automatic plant leaf disease identification using DenseNet-121 architecture with a mutation-based henry gas solubility optimization algorithm.  Neural Computing and Applications.  34 (7), 5513–5534 (2022).
  37. Tan M., Le Q. V.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.  Preprint arXiv:1905.11946 (2019).
  38. FER-2013, https://www.kaggle.com/datasets/msambare/fer2013.
  39. Facial Emotion Recognition Dataset, https://www.kaggle.com/datasets/tapakah68/facial-emotion-recognition.
  40. Face Expression Recognition Dataset, https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognitio....
  41. Hicham N., Karim S., Habbat N.  Customer sentiment analysis for Arabic social media using a novel ensemble machine learning approach.  International Journal of Electrical and Computer Engineering (IJECE).  13 (4), 4504–4515 (2023).