Researchers use graphene oxide to create a self-powered sensor that translates sign language into audio

Researchers from Tsinghua University recently developed a self-powered sensor that can monitor and detect multiple environmental stimuli simultaneously and demonstrated how it can “translate” sign language into audio.

The sensor was made from graphene oxide and powered internally by a moist electric generator called MEG, which contains a membrane that spontaneously absorbs water from the air. When water adheres to the surface, this results in a higher concentration of hydrogen ions at the top of the membrane and a potential difference between its two electrodes.

“The sensor develops a positive and negative charge separation and potential after adsorbing water in air, like the spontaneously bioelectrical potential generation in human skin,” explained Liangti Qu, a professor in the Department of Chemistry & Department of Mechanical Engineering at Tsinghua University and the study’s author. “When stimulated by ambient humidity, temperature, pressure, and light, the device produces electrical charges that induce a potential variation, producing distinct response signals to these stimuli.”

The team revealed that the development process was assisted by a machine learning module, which helped to combine multiple external responses into a single signal that the device can then learn, store, and interpret.

In the study, Qu and his colleagues Huhu Cheng and Ce Yang, attached the sensor to a volunteer’s wrist and set about teaching the machine learning algorithm linked to the sensor to read and translate different finger and hand movements and associate them with different words and phrases.

“Different finger movements and hand gestures produce unique pressure stimulus sequences, which can be collected on a workstation via Bluetooth,” explained Cheng. “The machine learning model is able to extract features from moist-electric potential response signal sequences and predict the change in each stimulus at each time step based on the output of the LSTM network for that time step.

“By decoding and analyzing the received signal with a trained machine learning model, the classification of gestures based on pressure sequences is possible. The gesture command and the corresponding audio is then showed on a smartphone in an app we developed.”

Beyond this, the researchers foresee a number of applications, including integration into the Internet of Things, health monitoring, human-computer interactions, and more. However, the technology is still nascent and a few challenges need to be overcome before widespread use could be possible.

“Environmental disturbances in complex application scenarios will cause noise in the response signal, which could be solved through the optimization of electrical signal processing or device performance,” said Qu. “We also need more training data. The distribution of the experimental data collected during tests and the actual application data will be different. We need to ensure the consistency of the training data and the test data.”

Posted: Oct 20,2022 by Roni Peleg