CN118116553A - Software and hardware combined hearing-impaired children music perception product based on gesture recognition technology - Google Patents

Abstract

The invention discloses a hearing-impaired children music perception product based on the combination of software and hardware of a gesture recognition technology, which has the characteristics that: the test grading module is used for grading the rehabilitation stage of the user to obtain a test grading result; the hardware module is used for performing music perception training of the user according to the test grading result; the interactive game module is used for playing a somatosensory game; and the personalized design module is used for performing DIY music creation. According to the embodiment, gesture recognition is realized by adopting a scheme combining Unity and OpenCV, mediaPipe, vibration signal conversion of music and sound is realized by adopting an Arduino hardware device, a music learning mode of hearing impaired children is created, pain point requirements of the hearing impaired children are converted into functional ideas in design practice, the hearing impaired children in a middle-high-level rehabilitation stage are helped to learn to perceive rhythm through a music somatosensory game design combining software and hardware, and the children can better feel and enjoy the music through a somatosensory rhythm game, so that physical and mental development of the children is promoted.

Description

A music perception system for hearing-impaired children based on a combination of hardware and software based on gesture recognition technology Product

Technical Field

The present invention relates to the field of hearing-impaired rehabilitation, and in particular to a music perception product for hearing-impaired children that combines software and hardware based on gesture recognition technology.

Background technique

With the development of the cause of the disabled in China, the importance of paying attention to the physical and mental health of special children has become increasingly prominent. As a group in urgent need of social support, hearing-impaired children are currently limited by factors such as domestic education level and social environment. The development of topics focusing on the mental health and interest and specialty cultivation of hearing-impaired children still needs to be strengthened.

Therefore, they need special education and rehabilitation measures to help them integrate into society. In addition to language rehabilitation, music education is also very important for their growth, because music does not require language to express emotions, and can help them better understand and feel the sounds around them. Especially for hearing-impaired children in the intermediate and advanced rehabilitation stages, after they have basic speech expression and cognitive abilities, they have the opportunity to conduct richer perceptual training. However, the current research and development of domestic music perception training systems is still imperfect, and there is no comprehensive music training game platform.

Most of the more common music products or applications on the market are designed for hearing people, such as traditional rhythm music games such as "Taiko no Tatsujin" and "Rhythm Master". They are difficult, have complex rhythms, are mainly for entertainment and competition, lack educational features, and have too complicated visual styles. It is difficult for hearing-impaired children to focus on both the pictures and the songs at the same time. There are also products that use feedback systems to enable hearing-impaired people to feel sounds, such as some music visualization and touchable devices. Currently, most existing music products for hearing-impaired people focus solely on the field of music visualization and have poor playability. At the same time, it is rare to design multiple solutions for products based on the level of hearing impairment. Although learning products for hearing children have fully considered the combination of content and fun, and also pay attention to personalized needs, there are relatively few music perception products designed specifically for hearing-impaired children.

Summary of the invention

The present invention is made to solve the above-mentioned problems, and aims to provide a music perception product for hearing-impaired children that combines software and hardware based on gesture recognition technology.

The present invention provides a music perception product for hearing-impaired children that combines software and hardware based on gesture recognition technology, and has the following characteristics: a test grading module, which is used to grade the user's rehabilitation stage and obtain a test grading result; a hardware module, which performs music perception training on the user according to the test grading result; an interactive game module, which is used to play somatosensory games; and a personalized design module, which is used to create DIY music.

The music perception product for hearing-impaired children that combines hardware and software based on gesture recognition technology provided by the present invention may also have the following features: wherein the test grading module performs user grading according to rehabilitation training theory, the user's hearing age and basic registration information, and the test grading result is any one of the low-level rehabilitation stage, intermediate-level rehabilitation stage and advanced rehabilitation stage.

The music perception product for hearing-impaired children based on the hardware and software combination of gesture recognition technology provided by the present invention may also have the following features: wherein the test grading module is based on the auditory noise test (ANT), the picture word identification test (WIPI) or the evaluation tool of physiological indicators, reflecting the user's hearing status and ability level, and performing more detailed grading. In addition, a rough grading is performed according to the user's hearing age and other basic information. The test grading module is aimed at users who are younger or have unclear rhythm cognition, and performs simple perceptual teaching with the assistance of the visual interface in the hardware module.

The music perception product for hearing-impaired children that combines software and hardware based on gesture recognition technology provided by the present invention may also have the following features: the hardware module includes music visualization hardware and music tactile hardware, the music tactile hardware realizes vibration according to the rhythm of the music, and the music visualization hardware is used to present a corresponding music visualization interface synchronously with the rhythm of the music, and selects two effects: a colorful heartbeat-like shape and a softer wave shape, to achieve different fluctuation rates and amplitudes according to the pitch and frequency of the music.

In the hardware and software combined music perception product for hearing-impaired children based on gesture recognition technology provided by the present invention, it can also have the following features: wherein, the music visualization hardware and the music touchable hardware are jointly produced using TouchDesigner and Arduino, and the specific implementation process is: Step 1, import the audio file and connect it with the analysis tool in TouchDesigner, and complete it by using nodes such as channel operator (CHOP) and Spectrum CHOP; Step 2, SpectrumCHOP will perform fast Fourier analysis on the audio and output the energy value within the corresponding frequency range; Step 3, use various nodes to convert the energy value into a visualization effect, use a vertex shader to control the shape or color, and link it to the output of Spectrum CHOP, and finally adjust the speed of the animation, reverse the animation, etc., and use the output module to save the presentation result as a video or image.

The music perception product for hearing-impaired children that combines software and hardware based on gesture recognition technology provided by the present invention may also have the following features: the music-touchable hardware includes a plurality of movable rounded corners, all of which are arranged on the surface above the product, and extend and retract with different amplitudes and frequencies following the rhythm of the music, so that the user can perceive the rhythm of the music through the tactile sense of the palm. The specific implementation method is as follows: each rounded corner is provided with a corresponding stepper motor. When the music starts, the music spectrum is first analyzed by fast Fourier transform to obtain energy intensity values of different frequencies, and then these values are further processed to obtain the rhythm information of the music. Finally, the stepper motor is controlled to move according to the rhythm information, so that the rounded corner performs corresponding actions. The rhythm information at least includes the position and rhythm intensity of the beat.

The hardware and software combined music perception product for hearing-impaired children based on gesture recognition technology provided by the present invention may also have the following features: the interactive game module mainly includes three levels: distinguishing the sounds of musical instruments, understanding sound characteristics, and learning nursery rhymes and rhythms. In the level of understanding sound characteristics, an associative teaching method is used to help users learn music knowledge. Specifically, by using the height of the stairs to compare the height of the sound, or using the size of the object to compare the strength of the sound, the user can deeply understand and remember the sound characteristics.

The music perception product for hearing-impaired children that combines software and hardware based on gesture recognition technology provided by the present invention may also have the following features: in the interactive game module, human posture data and interactive command operation data are collected to evaluate the user's game performance, understand the user's operating habits and reaction speed, and use MediaPipe gesture recognition technology to obtain human posture data. The user's skeletal posture is captured by a camera and matched with a standard image. According to the requirements of the game or problem, the interactive command operation data generated by the user when completing the operation is collected.

The hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology provided by the present invention may also have the following features: wherein the gesture recognition technology uses a human gesture recognition method based on MediaPipe and OpenCV libraries, and outputs recognition results of corresponding actions according to some predefined human actions. The specific implementation process is as follows: Step A, preprocessing the data, the preprocessing steps include video acquisition, video frame separation, and cropping of each frame of the image to extract human body parts, decomposing the video into image frames, and using the "CascadeClassifier" in the OpenCV library to detect the face position, and crop the image frames around the face position to ensure that background noise is reduced and only human body parts are retained; Step B, using the MediaPipe library to detect and locate key points of the human body, the MediaPipe library uses a pre-trained machine learning model to convert the input image frame into a human skeleton, detect and output the position information of key points of the human body, these key points include the head, shoulders, elbows, wrists, waist, knees and ankles, etc., by detecting and calculating the angles, distances and other attributes between different key points, a specific human gesture is identified.

The hardware-software-based music perception product for hearing-impaired children provided by the present invention based on gesture recognition technology may also have the following features: wherein, the appearance of the hardware-software-based music perception product for hearing-impaired children uses dinosaurs, turtles and other animal mimicry as the prototype of the shape, uses a rounded design style, and the size is designed to be within 20cm*20cm*20cm. The working process of the hardware-software-based music perception product for hearing-impaired children based on gesture recognition technology is as follows: Step 1, the test grading module determines the user's hearing impairment level through the speech audiometry link; Step 2, if the user's hearing impairment level is in the middle or low-level rehabilitation stage, first feel the rhythm of the music through the hardware module, and understand the concept and knowledge of rhythm. If it is in the advanced rehabilitation stage, skip the hardware module; Step 3, the user enters the interactive game module, first enters the functions of song selection, user settings, novice tutorials, and then enters the level challenge; Step 4, the user performs DIY song creation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram of the solution feedback system architecture in this embodiment.

FIG. 2 is a schematic diagram of the composition and functions of the hardware system in this embodiment.

FIG. 3 is a diagram of a low-fidelity prototype of hardware product components in this embodiment.

FIG. 4 is a diagram of the hardware product in this embodiment.

FIG. 5 is a music visualization interface produced by TouchDesigner in this embodiment.

FIG. 6 is a diagram showing the architecture of the music perception game system in this embodiment.

FIG. 7 is a schematic diagram of a storyboard of a user using a music game.

FIG8 is a low-fidelity prototype diagram of the rhythm game interface in this embodiment.

FIG. 9 is a schematic diagram of the gesture recognition technology in this embodiment.

FIG. 10 is a schematic diagram of an actual scenario when a user uses software and hardware products in this embodiment.

FIG. 11 is a flowchart of product usage for hearing-impaired children at different rehabilitation stages in this embodiment.

Detailed ways

In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is specifically described in the following embodiments in conjunction with the accompanying drawings.

This embodiment combines domestic and foreign music education methods, rehabilitation training theories for hearing-impaired children, and the application of music teaching methods in a rehabilitation center to diverge concepts and methods, adopts a solution combining Unity, OpenCV, and MediaPipe to realize posture recognition, and adopts Arduino hardware devices to realize the vibration signal conversion of music and sound, innovates the music learning method of hearing-impaired children, and transforms the pain points of hearing-impaired children into functional concepts in design practice. Through the design of music somatosensory games combining software and hardware, it helps hearing-impaired children in the middle and advanced rehabilitation stages learn to perceive rhythm, and through somatosensory rhythm games, let children better feel and enjoy music to promote their physical and mental development. This embodiment not only includes tactile hardware and visual interfaces, but also utilizes the characteristics of music to provide hearing-impaired children with a unique and interesting way of learning and development.

FIG1 is a diagram of the solution feedback system architecture in this embodiment.

The main goal of this product is to enable hearing-impaired children to understand and master the characteristics of sound, music rhythm, certain cognitive abilities, and body coordination abilities. The main roles are parents and hearing-impaired children. Software and hardware products can give users feedback information through visual output, tactile output, auditory output, etc. Therefore, as shown in Figure 1, the hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology in this embodiment includes a test grading module, a hardware module, an interactive game module, and a personalized design module.

The test grading module is used to grade the user's rehabilitation stage and obtain test grading results. In this embodiment, expert interviews and field user surveys were conducted among teachers and hearing-impaired children to obtain important demand points for the physical and mental characteristics of hearing-impaired children and the realization of perceptual training. First, users are graded according to rehabilitation training theory, the user's hearing age, and basic registration information, and the test grading results obtained include low-level rehabilitation stage, intermediate rehabilitation stage, and advanced rehabilitation stage. Compared with hearing-impaired children in other stages, hearing-impaired children in intermediate and advanced rehabilitation stages have improved hearing and cognitive abilities, and their demand for assistive products is more obvious.

In addition, the test grading module also reflects the user's hearing status and ability level based on ANT, WIPI or physiological indicator assessment tools, and provides more detailed grading. In addition, a rough grading is performed based on the user's hearing age and other basic information. Finally, for users who are younger or have unclear rhythm cognition, the test grading module provides simple perceptual teaching with the assistance of the visual interface in the hardware module.

The hardware module conducts music perception training for users based on the test grading results. When conducting music training, the special needs of hearing-impaired children need to be taken into consideration. According to research, the auditory system of hearing-impaired people may be more sensitive to low-frequency sounds, so low-frequency, soothing music that is hearing-impaired-friendly is selected as the initial challenge track. Choosing such music ensures that everyone can feel the pleasure and relaxation brought by music without missing any details due to hearing problems.

For hearing-impaired children, in addition to the auditory channel, visual observation, feeling rhythm and vibration can help them appreciate music. In order to ensure that these children can hear the sound or vibration of music and conduct effective training and experience, multi-sensory channel input is the best choice. Therefore, this embodiment uses visual and tactile stimulation at the same time, which can better help hearing-impaired children perceive and understand the rhythm and rhythm of music.

FIG. 2 is a schematic diagram showing the composition and functions of the hardware system in this embodiment.

As shown in Figure 2, in this embodiment, the hardware module includes music visualization hardware and music tactile hardware. The music tactile hardware can help hearing-impaired children understand the rhythm of music through tactile experience and achieve vibration according to the rhythm of music. When the music tactile function is not used, the product can be used as a music player with a cute shape, and some well-known children's songs are stored inside. In addition to the basic music playback function, the toy can also achieve vibration according to the rhythm of music, which enables users to feel the rhythm and beat of music through touch. When the vibration function is used, the software platform can also synchronously present the corresponding music visualization interface. These visualization interfaces not only allow users to perceive the waveform and spectrum of music, but also allow them to feel the rhythm and rhythm of music.

FIG. 3 is a diagram of a low-fidelity prototype of hardware product components in this embodiment.

In this embodiment, the music visualization hardware and the music touchable hardware are produced by using TouchDesigner and Arduino in conjunction with each other. The specific implementation process is as follows:

Step 1. TouchDesigner is a visual programming environment that can be used to create music visualizations. Import audio files and connect them with analysis tools in TouchDesigner by using nodes such as Channel Operators and Spectrum CHOP.

Step 2: Spectrum CHOP will perform fast Fourier analysis on the audio and output the energy value in the corresponding frequency range.

Step 3, use various nodes to convert energy values into visualization effects, use vertex shaders to control shape or color, and link it to the output of Spectrum CHOP, and finally adjust the speed of the animation, reverse the animation, etc., and use the output module to save the rendering results as a video or image.

In terms of the shape of hardware products, since the target users are children, it is necessary to consider its safety, age-appropriateness, shape and functionality. Children are sensitive to the appearance of products, and the shape of toys should be simple and easy to understand, easy to attract children's attention, and should not have sharp corners, fragile parts or small parts that may cause harm to children.

FIG. 4 is a diagram of the hardware product in this embodiment.

Therefore, as shown in FIG4 , this embodiment uses animal mimicry such as dinosaurs and turtles as prototypes of the shape, uses a rounded design style, and the size is designed to be within 20cm*20cm*20cm. Children can hold it with both hands or put one hand on the top of the product to feel the vibration. It can be close to children and satisfy their desire to explore.

Since the essence of sound is vibration. A music vibration device is a device that converts music signals into mechanical vibrations. In addition to stimulating users to perceive the rhythm of music from the visual channel, this embodiment also designs a hardware product to stimulate users from the tactile channel by generating vibrations of different amplitudes according to the rhythm of music. Therefore, this embodiment designs a plurality of movable fillets, which are arranged on the surface above the product as the parts for realizing the functions, and extend and retract with different amplitudes and frequencies according to the rhythm of music, so that users can perceive the rhythm of music through the sense of touch of the palm.

The specific implementation method is: each rounded corner is equipped with a corresponding stepper motor. When the music starts, the music spectrum is first analyzed by fast Fourier transform to obtain the energy intensity values of different frequencies, and then these values are further processed to obtain the rhythm information of the music. Finally, the stepper motor is controlled according to the rhythm information, so that the rounded corner makes the corresponding action. The rhythm information is the position and rhythm intensity of the beat.

FIG. 5 is a music visualization interface produced by TouchDesigner in this embodiment.

As shown in Figure 5, in the picture design of music visualization, combined with the hardware modeling, the music visualization hardware presents the corresponding music visualization interface through the synchronization of music rhythm, and selects two effects: a brightly colored heartbeat-like shape and a softer wave shape. Different fluctuation rates and amplitudes are achieved according to the pitch and frequency of the music, which visually stimulates hearing-impaired children and makes it easier for them to understand the characteristic of sound - rhythm.

The interactive game module is used to play somatosensory games. In this embodiment, the level grading with increasing difficulty and the step-by-step game design are designed. Such a game design can help hearing-impaired children gradually improve their skill level and will not lose interest due to excessive difficulty. In addition, some visual elements and symbols are used in the game to enhance information transmission and help hearing-impaired children better understand the game content. Through meticulous game design and level settings that conform to the laws of rehabilitation of hearing-impaired children, the playability and educational nature of the game can be improved, and the rehabilitation needs of hearing-impaired children can also be better met.

FIG. 6 is a diagram showing the architecture of the music perception game system in this embodiment.

As shown in FIG6 , in this embodiment, the interactive game module mainly includes three levels: identifying the sound of musical instruments, understanding sound characteristics, and learning nursery rhymes and rhythms. In the level of understanding sound characteristics, an associative teaching method is used to help users learn music knowledge. Specifically, by using the height of the stairs to compare the height of the sound, or using the size of the object to compare the strength of the sound, the user can deeply understand and remember the sound characteristics. This heuristic teaching method not only helps hearing-impaired children better understand and remember knowledge, but also increases the fun of the game.

FIG. 7 is a schematic diagram of a storyboard of a user using a music game.

FIG8 is a low-fidelity prototype diagram of the rhythm game interface in this embodiment.

As shown in Figures 7 and 8, take the somatosensory rhythm game level as an example: after clicking to start the game, there will be a tutorial for beginners. The sound button and text information in the window help players understand the gameplay. Players follow the graphic instructions and there will be interactive special effects feedback after success. Users make hip-akimbo, stomping and hand-raising movements at the corresponding nodes according to the drum beats of the song and the visual guidance on the interface. After the game, there are game data and positive feedback to the user.

In addition, the interactive game module collects human posture data and interactive command operation data to evaluate the user's game performance and understand the user's operating habits and reaction speed.

In this embodiment, MediaPipe posture recognition technology is used to obtain human posture data, and the user's skeletal posture is captured by the camera and matched with the standard image. These data are used for visual feedback, and the user's action skills and habits can be understood by analyzing these data in the future, providing a reference for optimizing the product's action recognition and feedback mechanism.

In terms of interactive instruction operation data, according to the requirements of the game or question, the interactive instruction operation data generated by the user when completing the operation is collected.

The above-mentioned posture recognition technology has been widely used in the field of computer vision, mainly for identifying human movements and postures. In this embodiment, a human posture recognition method based on MediaPipe and OpenCV library is used, and the recognition results of corresponding movements are output according to some predefined human movements. It mainly recognizes human postures and movements by detecting the positions of key points of human postures and calculating the distances and angles between different key points, and performs corresponding applications. The specific implementation process is as follows:

Step A, preprocessing the data, the preprocessing steps include video acquisition, video frame separation, and cropping each frame to extract the human body parts. The video is decomposed into image frames, and the "CascadeClassifier" in the OpenCV library is used to detect the face position and crop the image frame around the face position to ensure that the background noise is reduced and only the human body parts are retained.

Step B, use the MediaPipe library to detect and locate the key points of the human body. The MediaPipe library provides a pose estimator that can detect the key points of human poses in each frame of video. The MediaPipe library uses a pre-trained machine learning model to convert the input image frame into a human skeleton, detect and output the location information of the key points of the human body, including the head, shoulders, elbows, wrists, waist, knees, and ankles, etc., and recognize specific human poses by detecting and calculating the angles, distances, and other attributes between different key points.

FIG. 9 is a schematic diagram of the gesture recognition technology in this embodiment.

In this embodiment, when making music in Unity, ButtonController is used to control the display before and after the beat is hit, and different colors are used to distinguish them. The song is divided according to each bmp value, and the time point when the hit operation needs to be performed is calculated according to the beat. The GameManager script is needed to process the output of whether the hit is successful, which are public void NoteHit(){Debug.Log("Hit")；} and public void NoteMiss()

{Debug.Log("Miss");}. In the Hierarchy view, you also need to add a UI-text component and name it Score to record the score. Finally, to integrate MediaPipe with Unity, use Unity's IronPython plug-in, display the camera image in Unity by calling UnityAPI, and use the "Mono.Data.Sqlite" library (for reading and writing SQLite database) for gesture recognition and music game production.

Finally, a personalized design module is used for DIY music creation. When the user can accurately select the correct sound characteristics and enter the somatosensory rhythm game, he can understand the rules of the game through the novice tutorial and guidance prompts. In this link, children can learn how to move according to the rhythm, and may also try to create their own unique rhythmic movements later, which increases the freedom and fun of the game. This embodiment designs an independent music creation DIY section, allowing users to freely splice notes and syllables to create personalized music works. This can not only increase children's interest and participation in music games, but also stimulate their creativity and potential in the field of music.

FIG. 10 is a schematic diagram of an actual scenario when a user uses software and hardware products in this embodiment.

FIG. 11 is a flowchart of product usage for hearing-impaired children at different rehabilitation stages in this embodiment.

As shown in FIG. 10 and FIG. 11 , the use process of the hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology in this embodiment is as follows:

Step 1: The test grading module determines the user's hearing impairment level through speech audiometry.

Step 2: If the user's hearing impairment level is at the low to medium rehabilitation stage, he or she can first feel the rhythm of music through the hardware module and understand the concepts and knowledge about rhythm. For users at the advanced rehabilitation stage, if they show good cognition in the initial side-hearing test, they can choose to skip the hardware interaction stage and directly enter the advanced somatosensory game stage.

Step three, the user enters the interactive game module, first enters the song selection, user settings, novice tutorial and other functions, and then enters the level challenge.

Step 4: Users create their own songs.

This embodiment uses auditory, visual, and tactile multimodal input and gamified interactive forms for hearing-impaired children in the target rehabilitation stage to maximize the possibility of letting children perceive music. Through technical means such as posture recognition and speech audiometry, the rhythm perception training and music teaching of hearing-impaired children are made daily through games, and the children's music creation ability is trained, so that children's interests and specialties can be cultivated while learning through entertainment, allowing hearing-impaired children to enjoy the same colorful childhood as normal-hearing children.

First, let the hearing-impaired children understand the sound characteristics they hear. Hearing-impaired children face many challenges in learning and development due to their hearing defects. One of the main difficulties is difficulty in understanding sound characteristics and game rules. It becomes more difficult for children with hearing loss to identify sound characteristics including volume, rhythm, etc., which will affect the development of their language learning and communication skills. In addition, they may not be able to hear the background noise when other people are talking, resulting in misunderstandings or incomplete understanding of the conversation. Understanding the rules of the game is also a challenge, because the rules of the game usually need to be understood through verbal instructions or listening to other people's explanations. Hearing-impaired children may need additional support, such as sign language, images, text instructions, etc., to help them understand the rules of the game and participate in social interactions.

Secondly, we need to achieve child-friendly design and improve user experience. Children’s reaction and body coordination abilities are relatively weak, and they are prone to making mistakes or losing patience. At the same time, children lack sufficient self-protection ability, and using dangerous shapes and game play may pose a safety threat to children.

Therefore, the product in this embodiment has the following advantages:

(1) Hearing-impaired children need music education because it helps fill their hearing gaps and improves their self-confidence, expression and comprehension through music appreciation, helping them release negative emotions and eliminate loneliness and autistic behavior. Music education can help hearing-impaired children develop comprehensively both physically and mentally, thus creating a better future for them. This music perception training product helps hearing-impaired children promote the coordinated development of their physical abilities, cultivate their sentiments through aesthetic education, and achieve their own comprehensive development.

(2) Use gamification to build a game platform for hearing-impaired children, improve the current single-format music teaching classroom in deaf schools, use digital means to "move" classrooms and courses to any place, make them part of daily life, and provide guidance to children in a more intuitive way. Combine technologies such as multimodal sensors and somatosensory games to innovate the interactive methods of music products for hearing-impaired children.

The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.

Claims (10)

1. A music perception product for hearing-impaired children based on gesture recognition technology and combining software and hardware, characterized by comprising: A test grading module is used to grade the user's rehabilitation stage and obtain a test grading result; A hardware module for performing music perception training on the user according to the test grading result; Interactive game module, used for playing somatosensory games; Personalized design module for DIY music creation. 2. The music perception product for hearing-impaired children based on gesture recognition technology and hardware and software according to claim 1 is characterized by: The test grading module performs user grading based on rehabilitation training theory, the user's hearing age and basic registration information. The test grading result is any one of a low-level rehabilitation stage, an intermediate-level rehabilitation stage, and a high-level rehabilitation stage. 3. The music perception product for hearing-impaired children based on gesture recognition technology and software and hardware integration according to claim 1 is characterized by: The test grading module reflects the user's hearing status and ability level based on the auditory noise test, image word recognition test or physiological index assessment tools, and performs more detailed grading. In addition, it performs a rough grading based on the user's hearing age and other basic information. The test grading module is aimed at users who are younger or have unclear rhythm cognition, and performs simple perceptual teaching with the assistance of the visual interface in the hardware module. 4. The music perception product for hearing-impaired children based on gesture recognition technology and software and hardware integration according to claim 1 is characterized by: The hardware module includes music visualization hardware and music touchable hardware. The music-touchable hardware vibrates according to the rhythm of the music. The music visualization hardware is used to present a corresponding music visualization interface through synchronization with the music rhythm, and select two effects: a colorful heartbeat-like shape and a softer wave shape, to achieve different fluctuation rates and amplitudes according to the pitch and frequency of the music. 5. The music perception product for hearing-impaired children based on gesture recognition technology and hardware and software according to claim 4 is characterized by: The music visualization hardware and the music touchable hardware are produced by using TouchDesigner and Arduino in conjunction with each other, and the specific implementation process is as follows: Step 1, import the audio file and connect it with the analysis tools in TouchDesigner by using nodes such as Channel Operator and Spectrum CHOP; Step 2, Spectrum CHOP will perform fast Fourier analysis on the audio and output the energy value in the corresponding frequency range; Step 3, use various nodes to convert energy values into visualization effects, use vertex shaders to control shape or color, and link it to the output of Spectrum CHOP, and finally adjust the speed of the animation, reverse the animation, etc., and use the output module to save the rendering results as a video or image. 6. The hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology according to claim 4, characterized in that: The music-touchable hardware includes a plurality of movable rounded corners, all of which are arranged on the surface above the product, and extend and retract with different amplitudes and frequencies following the rhythm of the music, so that the user can perceive the rhythm of the music through the sense of touch of the palm. The specific implementation method is as follows: Each of the fillets is provided with a corresponding stepper motor. When the music starts, the music spectrum is first analyzed by fast Fourier transform to obtain the energy intensity values of different frequencies, and then these values are further processed to obtain the rhythm information of the music. Finally, the stepper motor is controlled to move according to the rhythm information, so that the fillets perform corresponding actions. The rhythm information includes at least the position and rhythm strength of the beat. 7. The hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology according to claim 1, characterized in that: The interactive game module mainly includes three levels: identifying the sound of musical instruments, understanding the sound characteristics, and learning nursery rhymes and rhythms. In the level of understanding sound characteristics, an associative teaching method is used to help users learn music knowledge. Specifically, by using the height of stairs to compare the height of sounds, or using the size of objects to compare the strength of sounds, users can deeply understand and remember the sound characteristics. 8. The hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology according to claim 1, characterized in that: In the interactive game module, human body posture data and interactive instruction operation data are collected to evaluate the user's game performance, understand the user's operation habits and reaction speed, The human body posture data is obtained by using MediaPipe posture recognition technology, and the user's skeleton posture is captured by a camera and matched with a standard image. According to the requirements of the game or question, the interactive instruction operation data generated by the user when completing the operation is collected. 9. The hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology according to claim 8, characterized in that: The posture recognition technology uses a human posture recognition method based on MediaPipe and OpenCV libraries, and outputs recognition results of corresponding actions according to some predefined human actions. The specific implementation process is as follows: Step A, preprocessing the data, the preprocessing steps include video acquisition, video frame separation, and cropping each frame of the image to extract the human body parts, decomposing the video into image frames, and using the "CascadeClassifier" in the OpenCV library to detect the face position and crop the image frame around the face position to ensure that the background noise is reduced and only the human body parts are retained; Step B, use the MediaPipe library to detect and locate the key points of the human body. The MediaPipe library uses a pre-trained machine learning model to convert the input image frame into a human skeleton, detect and output the location information of the key points of the human body, including the head, shoulders, elbows, wrists, waist, knees and ankles, etc., and recognize specific human postures by detecting and calculating the angles, distances and other attributes between different key points. 10. The hardware-software combined music perception product for hearing-impaired children based on gesture recognition technology according to claim 1, characterized in that: The appearance of the hearing-impaired children's music perception product based on gesture recognition technology, which combines software and hardware, uses animal mimicry such as dinosaurs and turtles as the prototype of the shape, uses a rounded design style, and the size is designed to be within 20cm*20cm*20cm. The working process of the music perception product for hearing-impaired children based on the combination of software and hardware based on gesture recognition technology is as follows: Step 1: The test grading module determines the user's hearing impairment level through speech audiometry; Step 2: If the user's hearing impairment level is at a low- to medium-level rehabilitation stage, the user will first use the hardware module to feel the rhythm of music and learn about the concept and knowledge of rhythm. If the user's hearing impairment level is at a high-level rehabilitation stage, the hardware module will be skipped. Step 3: The user enters the interactive game module, first entering the song selection, user settings, novice tutorial and other functions, and then enters the level challenge; Step 4: Users create their own songs.