Nanjing University Of Posts And Telecommunications | China

Zhang Jian | Huo YouLiang | Ran JunYao

Citing the World Federation of the Deaf (WFD) data, there are about 70 million deaf-mutes in the world, which is more than 1% of the world's total population. As a special group in society, deaf-mutes often rely on sign language to communicate and learn knowledge due to their own hearing impairment. At present, the most common way to learn sign language is to learn from lexicon teachers in special schools with the contents of sign language teaching materials. However, since the number of sign language teachers is far less than the number of deaf students, it is difficult for teachers to ensure teaching quality for each student. This shortage of educational resources is especially evident in underdeveloped areas. If an efficient sign language learning APP is set up to assist teachers in teaching and tailoring sign language learning programs to each and every student and to track the their learning quality so as to give consideration to everyone, it is a powerful weapon to the shortage of education resources. At present, there is currently a small amount of software for sign language learning on the market, and various problems exist in existing software. One of the most prominent problems is that existing sign language learning software can only display the hand movements, while neglecting the facial expressions and torso movements. It is the facial expressions and torso movements that play a key role in the expression of sign language emotion information. In addition, most sign language learning software is in the form of still pictures, teaching by tracing gestures which has poor user friendliness and difficult to learn sign language. To overcome the above shortcomings, the project will simulate human facial expressions and trunk movements for different scenes through the method of sentiment analysis. Combining with the hand movements, the project will eventually be presented in the form of animation to make the sign language expression approach the real effect. To sum up, this project intends to design a set of personalized deaf and mute teaching system, which consists of two parts: sign language teaching and speech-sign language translation.