The work primarily focuses on addressing the contemporary challenge of hand gesture recognition, driven by the overarching objectives of revolutionizing military training methodologies, enhancing human-machine interactions, and facilitating improved communication between individuals with disabilities and machines. In-depth scrutiny of the methods for hand gesture recognition involves a comprehensive analysis, encompassing both established historical computer vision approaches and the latest deep learning trends available in the present day.
This investigation delves into the fundamental principles that underpin the design of models utilizing 3D convolutional neural networks and visual transformers. Within the 3D-CNN architecture that was analyzed, a convolutional neural network with two convolutional layers and two pooling layers is considered. Each 3D convolution is obtained by convolving a 3D filter kernel and summing multiple adjacent frames to create a 3D cube. The visual transformer architecture that is consisting of a visual transformer with Linear Projection, a Transformer Encoder, and two sub-layers: the Multi-head Self-Attention (MSA) layer and the feedforward layer, also known as the Multi-Layer Perceptron (MLP), is considered.
This research endeavors to push the boundaries of hand gesture recognition by deploying models trained on the ASL and NUS-II datasets, which encompass a diverse array of sign language images. The performance of these models is assessed after 20 training epochs, drawing insights from various performance metrics, including recall, precision, and the F1 score. Additionally, the study investigates the impact on model performance when adopting the ViT architecture after both 20 and 40 training epochs were performed.
This analysis unveils the scenarios in which 3D convolutional neural networks and visual transformers achieve superior accuracy results. Simultaneously, it sheds light on the inherent constraints that accompany each approach within the ever-evolving landscape of environmental variables and computational resources.
The research identifies cutting-edge architectural paradigms for hand gesture recognition, rooted in deep learning, which hold immense promise for further exploration and eventual implementation and integration into software products.
1. Molchanov, P., Gupta, S., Kim, K., & Kautz, J. (2015). Hand gesture recognition with 3D convolutional neural networks.
https://doi.org/10.1109/CVPRW.2015.7301342
2. Molchanov, P., Gupta, S., Kim, K., & Pulli, K. (2015). Multi-sensor system for driver's hand-gesture recognition. 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 1, 1-8.
https://doi.org/10.1109/FG.2015.7163132
3. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014). Large-scale video classification with convolutional neural networks. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 223, 1725-1732.
https://doi.org/10.1109/CVPR.2014.223
4. Ohn-Bar, E., & Trivedi, M. M. (2014). Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations. IEEE Transactions on Intelligent Transportation Systems, 15, 2368-2377.
https://doi.org/10.1109/TITS.2014.2337331
5. Simonyan, K., & Zisserman, A. (2014). Two-stream convolutional networks for action recognition. https://doi.org/10.48550/arXiv.1406.2199
6. Tran, D., Bourdev, L. D., Fergus, R., Torresani, L., & Paluri, M. (2015). Learning spatiotemporal features with 3D convolutional networks. 2015 International Conference on Computer Vision, 9, 4489-4497.
https://doi.org/10.1109/ICCV.2015.510
7. Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2014). Multiscale deep learning for gesture detection and localization, 474-490.
https://doi.org/10.1007/978-3-319-16178-5_33
8. Yong, T., Kian, L., Connie, T., Chin-Poo, L., & Cheng-Yaw, L. (2021). Convolutional neural network with spatial pyramid pooling for hand gesture recognition. Neural Computing and Applications, 33, 1-13.
https://doi.org/10.1007/s00521-020-05337-0
9. Yong, T., Kian, L., & Chin-Poo, L. (2021). Hand Gesture Recognition via Enhanced Densely Connected Convolutional Neural Network. Expert Systems with Applications, 175.
https://doi.org/10.1016/j.eswa.2021.114797
10. Osimani, C.; Ojeda-Castelo, J. J.; & Piedra-Fernandez, J. A. (2023). Point Cloud Deep Learning Solution for Hand Gesture Recognition. International Journal of Interactive Multimedia and Artificial Intelligence.
https://doi.org/10.9781/ijimai.2023.01.001
11. Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics. https://doi.org/10.18653/v1 %2FN19-1423
12. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners.
13. Hengshuang, Z., Jiaya, J., & Vladlen, K. (2020). Exploring Self-Attention for Image Recognition. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10073-10082.
https://doi.org/10.1109/CVPR42600.2020.01009
14. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers.
https://doi.org/10.1007/978-3-030-58452-8_13
15. Ji, S. Xu, W., Yang, M., & Yu, K. (2010) 3 d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35 (1), 495-502.
https://doi.org/10.1109/TPAMI.2012.59
16. Barczak, A. L. C., Reyes, N. H., Abastillas, M., Piccio, A., & Susnjak, T. A. (2011). New 2D Static Hand Gesture Colour Image Dataset for ASL Gestures.
17. Pisharady, P. K., Vadakkepat, P., & Loh, A. P. (2013). Attention based detection and recognition of hand postures against complex backgrounds. International Journal of Computer Vision, 101, 403-419.
https://doi.org/10.1007/s11263-012-0560-5