Computer System for Converting Gestures to Text and Audio Messages

: cc. 90 - 97
Lviv Polytechnic National University
Lviv Polytechnic National University
Національний університет «Львівська політехніка», кафедра електронних обчислювальних машин

Today, there are quite a large number of deaf- mute and hard-of-hearing people which communicate using gestures. Therefore, it is simply necessary to provide them with modern means of communication with the surrounding world. This paper creates a holistic computer system architecture for converting gestures into text and audio messages. The principles of construction and basic design solutions of a computer system based on a modern element base with increased productivity and minimization of hardware costs and energy consumption have been developed. The most popular existing solutions for gesture recognition are considered and analyzed. The operation of the main components has been described, the principle of functioning of the entire system has been analyzed, and their advantages and disadvantages have been compared. The latest structural components for building a computer system (both physical and software) have been selected and investigated. Physical features include: the state-of-the-art Arduino Nano computing platform, the HC-05 Bluetooth module, the ADXL335 accelerometer, and the latest ZD10-100 Information sensor (flexibility sensor). Software features include: firmware for the Arduino Nano hardware platform, Python-based software for splitting the flow of letters into words, displaying them, and voicing them. The methods of Google Media Translation API and Google Text-to-speech (gTTS) have been analyzed. The expediency of conducting research has improved performance through the use of a new information sensor, which is a flexibility sensor ZD10-100 500 g. The general structural scheme of all systems has been designed.

  1. (2017). World and Ukrainian statistics of the deaf and dumb, International Day of the Deaf. [online] Available at: (Accessed: 5 October 2021)
  2. (2014). Description Enable talk system. [online] Available at: sistema-enable-talk-inejjbl-tolk.html (Accessed: 6 October 2021)
  3. (2018). Description NuGlove system, Interact system, Droid Agent system. [online] Available at: (Accessed: 6 October 2021)
  4. (2021). SignALL translate program. [online] Available at: (Accessed: 6 October 2021)
  5. D. Mheill. “Gesture and Thought”, University of Chicago Press, Degruyter, 2005. DOI: 10.7208/9780226514642.
  6. (2004). The bilingual method is an indispensable condition for the learning of the deaf. [online] Available at: (Accessed: 5 October 2021) 
  7. Y.  Badamasi.  “The  working  principle  of  an  Arduino”, ieeexplore, 2014. DOI: 10.1109/ICECCO.2014.6997578.
  8.          (2010).           Specification      analogue accelerometer ADXL335.                              [online]     Available   at: overview (Accessed: 7 October 2021)
  9. V. Sing, A. Shahy, A. Beg, B. Chay, S. Kymar. “The Specification Bluetooth HC-05 module”, ieeexplore, 2019 . DOI: 10.1109/ICACAT.2018.8933698.
  10.  (2021).  Thin  film  flex  and  pressure sensor         ZD10-100  500g         [online]                     Available   at: film-pressure-sensor-flex-bend-sensor-zd-10-100-500-g- resistance-type-fsr-sensor-thin-film-pressure-sensor-force- sensing-resistor-force-sensitive-resistor           (Accessed:              7 October 2021)
  11. J. Crammer. “Python -The Fastest Growing Programming Language”, International Research Journal of Engineering and Technology (IRJET), vol. 4, iss. 12, 2017. DOI: 2395-0072/pan.2017.12.
  12. (2021). Word segmentation module v1.3.1. [online] Available at: (Accessed: 8 October 2021)
  13. E. Vries, M. Schoonvelde, G. Schumacher. “Google Translate Works” Political Analysis, vol. 26, iss. 4, 2018, pp. 417 – 430. DOI: 10.1017/pan.2018.26.
  14. (2021). Google Text-to-speech module v2.2.3. [online] Available at: (Accessed: 8 October 2021)
  15. (2021). Google Media Translation API. [online] Available at: translation (Accessed: 8 October 2021).