CN114466298B - Sound control device based on multi-sensory data and control equipment using the same - Google Patents

Abstract

The invention discloses a sound regulating and controlling device based on multi-sense data, which can be connected with a computer, a hearing test device and a tablet computer, wherein the sound regulating and controlling device comprises a master controller, a data acquisition unit, a data operation unit and a sound automatic regulating and controlling unit, wherein the data acquisition unit, the data operation unit and the sound automatic regulating and controlling unit are connected with the master controller; the data acquisition unit comprises a data acquisition module and a feedback control module, and the data acquisition module is used for acquiring data generated by a subject in the hearing test process; the feedback control module is used for feeding back information and controlling the hearing test device to send out corresponding audio signals; the data operation unit is used for carrying out integrated analysis and operation on the data in the memory to form a multi-sense database and storing the multi-sense database in the memory; the sound automatic regulation and control unit can automatically regulate and control the sound signals output by the hearing test device according to the multisensory database. A regulating device using the sound regulating device based on the multi-sense data is also disclosed.

Description

Sound control device based on multi-sensory data and control equipment using the same

Technical Field

The present invention relates to the field of hearing equipment, and in particular to a sound control device based on multi-sensory data and a control device using the same.

Background technique

The volume adjustment of existing headphones does not distinguish between left and right ears, nor between frequency bands. When adjusting the volume, the volume of each frequency is adjusted at the same time, and both ears are adjusted at the same time, without considering the needs of users.

Existing hearing aid products can adjust the volume according to the hearing threshold data of the left and right ears, but the purpose of hearing aids is to amplify the sound or increase the gain. The volume control of hearing aids is to amplify the volume. This adjustment method can easily lead to excessive sound stimulation and harsh sound, resulting in a phenomenon where there are many sounds but they cannot be heard clearly. For the deaf, it is very frustrating that they could not hear any sound before, but after using hearing aids, they can hear more sounds but cannot hear them clearly.

For some people with hidden hearing loss and mild hearing loss, there is basically no communication barrier and they cannot accept hearing aids. However, the sound played by ordinary headphones or mobile phone speakers does not match the individual's hearing status. Loud sounds lead to further hearing loss, while soft sounds lead to insufficient stimulation and functional decline, which in turn causes tinnitus problems.

Each individual's physical condition and sensitivity are different, so how to set the sound entering the ear is related to the protection or maintenance of the auditory organs and functions for many people.

Summary of the invention

The purpose of the present invention is to provide a device that can integrate data such as "gesture data A, visual data E and auditory data H" to set and automatically control the volume of in-ear sound, that is, to provide a sound control device based on multi-sensory data, and also to provide a sound control device using the same, which can at least solve one of the above-mentioned problems.

According to one aspect of the present invention, a sound control device based on multi-sensory data is provided, which can be connected to a computer, a hearing test device, and a tablet computer. The sound control device includes a host and a master controller installed in the host, a data acquisition unit connected to the master controller, a data calculation unit, and a sound automatic control unit; the data acquisition unit includes a data acquisition module and a feedback control module, the data acquisition module can be installed in the computer, and is used to collect gesture data A, visual data E, and hearing data H generated by the subject during the hearing test process; the feedback control module can be installed in the host, and is used to feedback information and control the hearing test device to emit a corresponding audio signal; the data calculation unit includes a data calculation module, a memory, and a counter, the memory is used to store the data collected by the data acquisition module, the counter is used to record the data, and the data calculation module is used to integrate, analyze, and calculate the data in the memory to form a multi-sensory database and store it in the memory; the sound automatic control unit includes a DSP, one end of the DSP is electrically connected to the master controller, and the other end is connected to the hearing test device, and the DSP can automatically control the sound signal output by the hearing test device according to the multi-sensory database.

Therefore, the present invention provides a sound control device based on multi-sensory data with a new structure, which can be connected and used in conjunction with hearing test devices such as computers, tablet computers and audiometers. The working principle is: when the computer controls the hearing test device to send a test audio signal, the subject receives the audio signal through the sound-generating device worn, observes the screen of the tablet computer with his eyes, finds the text signal corresponding to the heard audio signal and responds by clicking the corresponding text information on the screen to form a response action. The first acquisition device records the subject's gestures and vision and feeds them back to the sound control device. The sound control device integrates and analyzes these data with the hearing data and related data and saves the analysis results.

The present invention designs a system that can collect and record a person's multi-sensory data in real time. The system can at least simultaneously collect and record gesture, vision, hearing and other data as well as related data, and use a signal to stimulate and simultaneously obtain gesture, vision, hearing and other response data, and integrate and analyze these data to obtain multi-sensory data; more importantly, the device can also perform calculations based on multi-sensory data to form a three-dimensional "gesture-vision-hearing" database, and the DSP can automatically adjust the sound signal output by the hearing test device according to the data in the database.

The sound control device based on multi-sensory data of the present invention uses a special hardware architecture to establish a digitization process of multi-sensory input signal response data, reintegrates the data ignored by traditional hearing tests into a three-dimensional data set, and finely controls the sound input to the human ear based on the data set, while meeting the multiple needs of listening to speech and appreciating music.

Compared with the traditional hearing threshold data format, which is limited to the annotation of hearing status, the present invention can establish a new hearing data set, which can meet the needs of scientific research on auditory perception characteristics, prevention of auditory diseases, and detection of changes in brain cognitive status;

The data of the adjusted in-ear sound signal will be stored in the master controller, which can be directly installed in products such as hearing aids or headphones. It can be used directly after assembly and is suitable for individuals with different hearing conditions, effectively making up for the shortcomings of existing headphones or hearing aids in sound control.

In some embodiments, the tablet computer includes a touch screen, the touch screen is electrically connected to the feedback control module, and is configured to display text information and provide the subject with a response operation; the feedback control module can control the text information to be displayed at different positions on the touch screen and record the position where the subject clicks the touch screen, the number of times the touch screen is clicked, and the time of contact with the touch screen;

The text information includes text signals corresponding to the audio signals emitted by the hearing test device and other text signals that do not correspond to the audio signals emitted by the hearing test device;

The main controller can control the text signal to be displayed at different positions of the touch screen.

As a result, the subject's visual data is recorded while watching the touch display screen, gesture data is recorded during the operation process, and auditory data is recorded during the test. These data are acquired in real time during the test and will not interfere with the test process. Moreover, these data are generated based on the subject's instinctive reaction, with high accuracy and high reference value.

In some embodiments, a loudness level meter and a volume increase/decrease button are also displayed on the touch display screen. The loudness level meter includes four levels: quiet S, medium M, loud L, and noisy N. The loudness step range of the loudness level meter is 5-10 dB. The volume increase/decrease button is used to control the output sound volume of the hearing test device. The loudness level meter allows the subject to determine the autonomously perceived loudness level and the data is recorded by the counter and stored in the memory. Thus, the loudness level meter and the volume increase/decrease button are displayed on the touch display screen. As the volume increases, the subject can determine which level of the loudness level meter the autonomously perceived loudness level is. After confirmation, the data is recorded by the counter and stored in the memory.

In some embodiments, the gesture data A includes at least the number of times the touch screen is clicked and the time interval between clicks of the touch screen; the visual data E includes at least location information, volume increase and decrease buttons, a loudness level table, etc.; the auditory data H includes at least audio data, loudness data, and spectrum data related to text information.

In some embodiments, the data calculation module first establishes a multi-sensory database based on the gesture data A, the visual data E, and the auditory data H, and then establishes a data table based on the data in the multi-sensory database. The data table can at least reflect the functional relationship between frequency and gesture, the functional relationship between loudness increment and gesture, and the functional relationship between frequency, loudness and gesture.

In some embodiments, the multi-sensory database obtained by the data calculation module is stored in the memory as the basic data for the DSP to automatically control the volume. The basic data of the multi-sensory database at least includes loudness level values of different frequencies, loudness increment response time values of different frequencies, random position change response accuracy, and the difference between the number of clicks and the number of sounds.

In some embodiments, the data acquisition unit records data related to clarity in the position identification of the speech signal, records data related to sensitivity in the manual time information after the stimulation sound, and records data related to the volume increment ΔP in the number of key presses of the volume increase and decrease control. These data form a three-dimensional "gesture-vision-hearing" data set, which is stored in a computer database like a conventional audiogram.

In some implementations, the DSP is embedded with an algorithm table that can process the voice signal and music signal of the ear sound separately. The purpose of the voice signal is to improve or maintain the clarity; the purpose of the music signal is to make it pleasant to the ear; the algorithm table includes two types: a voice mode algorithm table and a music mode algorithm table; the DSP can automatically switch to the voice mode or the music mode according to the input signal. Therefore, different modes can be switched according to different sound signals to meet the use requirements.

In some implementations, the DSP can automatically configure the dynamic sound pressure range ΔD and the incremental value ΔP based on the data in the multi-sensory database in conjunction with the algorithm table, automatically configure the frequency band range ΔF according to the input signal, and store the dynamic sound pressure range ΔD, the incremental value ΔP, and the frequency band range ΔF in the form of a data packet in the built-in RAM; each incremental item in the data packet is stored in a format not less than n orders, and the n value can be modified and expanded; the control parameters set for the voice mode and the music mode are different, and the control parameters are configured in the form of a data packet. In this way, a larger range of hearing thresholds can be covered, enhancing applicability.

According to another aspect of the present invention, there is also provided a sound control device based on multi-sensory data, comprising at least:

computer;

an auditory testing device connected to the computer and configured to generate an audio signal;

a sound-generating device connected to the hearing test device and configured to transmit an audio signal to the ear of the subject;

a sound control device connected to the computer and the hearing test device and configured to generate a text signal based on the audio signal;

A tablet computer, connected to the sound control device, configured to display text information and allow the subject to respond and operate;

A first collection device, connected to the sound control device, configured to collect the subject's actions and vision during the response process;

The sound control device is the above-mentioned sound control device;

The data acquisition module is installed on the computer and is used to collect the gesture data A, visual data E, and auditory data H generated by the subject during the test.

Therefore, the present invention provides a sound control device based on multi-sensory data with a new structure. The device applies the above-mentioned sound control device. The working principle of the device is: when the computer controls the hearing test device to send out a test audio signal, the subject receives the audio signal through the sound-generating device worn, observes the screen of the tablet computer with his eyes, finds out the text signal corresponding to the heard audio signal and responds by clicking the corresponding text information on the screen to form a response action. The first acquisition device records the subject's gestures and vision and feeds them back to the sound control device. The sound control device integrates and analyzes these data with the hearing data and related data and saves the analysis results.

This device can at least simultaneously collect and record gesture, vision, hearing and other data as well as related data, use one signal to stimulate and simultaneously obtain gesture, vision, hearing and other response data, and integrate and analyze these data to obtain multi-sensory data; more importantly, this device can also perform calculations based on multi-sensory data to form a three-dimensional "gesture-vision-hearing" database, and the DSP can automatically adjust the sound signal output by the hearing test device according to the data in the database.

In some embodiments, the multi-sensory data-based sound control device further includes a second acquisition device, which is electrically connected to the master controller and configured to collect the voices emitted by the subject during the test. Thus, some subjects may emit sounds during the test, and the second acquisition device may collect the sounds emitted by the subjects and feed them back to the sound control device, which will analyze the data with other relevant data.

Beneficial effects of the present invention:

The present invention designs a system that can collect and record a person's multi-sensory data in real time. The system can at least simultaneously collect and record gesture, vision, hearing and other data as well as related data, and use a signal to stimulate and simultaneously obtain gesture, vision, hearing and other response data, and integrate and analyze these data to obtain multi-sensory data; more importantly, the device can also perform calculations based on multi-sensory data to form a three-dimensional "gesture-vision-hearing" database, and the DSP can automatically adjust the sound signal output by the hearing test device according to the data in the database.

The sound control device based on multi-sensory data of the present invention uses a special hardware architecture to establish a digitization process of multi-sensory input signal response data, reintegrates the data ignored by traditional hearing tests into a three-dimensional data set, and finely controls the sound input to the human ear based on the data set, while meeting the multiple needs of listening to speech and appreciating music.

Compared with the traditional hearing threshold data format, which is limited to the annotation of hearing status, the present invention can establish a new hearing data set, which can meet the needs of scientific research on auditory perception characteristics, prevention of auditory diseases, and detection of changes in brain cognitive status;

The data of the adjusted in-ear sound signal will be stored in the master controller, which can be directly installed in products such as hearing aids or headphones. It can be used directly after assembly and is suitable for individuals with different hearing conditions, effectively making up for the shortcomings of existing headphones or hearing aids in sound control.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a simplified block diagram of a sound control device based on multi-sensory data according to an embodiment of the present invention;

FIG2 is a structural block diagram of a further refined version of the sound control device based on multi-sensory data shown in FIG1 ;

FIG3 is a simplified top view of the touch display screen of a tablet computer in one usage scenario according to an embodiment of the present invention;

FIG4 is a simplified three-dimensional schematic diagram of a sound control device based on multi-sensory data according to an embodiment of the present invention;

FIG5 is a simplified block diagram of a sound control device based on multi-sensory data according to an embodiment of the present invention;

FIG. 6 is a further refined structural block diagram of the sound control device based on multi-sensory data shown in FIG. 5 .

The reference numerals in FIGS. 1 to 6 are as follows: 1-computer; 2-audio testing device; 3-sound generating device; 4-sound control device; 5-tablet computer; 6-first acquisition device; 7-second acquisition device; 40-host; 41-general controller; 42-data acquisition unit; 43-data calculation unit; 44-sound automatic control unit; 45-communication module; 46-PCB board; 47-port; 51-touch screen; 421-data acquisition module; 422-feedback control module; 431-data calculation module; 432-memory; 433-counter; 441-DSP; 511-loudness level meter; 512-volume increase and decrease button.

Detailed ways

The present invention will be further described in detail below in conjunction with the specific drawings.

1 to 4 schematically show a sound control device based on multi-sensory data according to an embodiment of the present invention.

As shown in FIGS. 1 to 4 , the sound control device 4 based on multi-sensory data can be connected to a computer 1 , a hearing test device 2 , and a tablet computer 5 .

The sound control device 4 includes a host 40 , a master controller 41 installed in the host 40 , a data acquisition unit 42 connected to the master controller 41 , a data calculation unit 43 , and a sound automatic control unit 44 .

The data acquisition unit 42 includes a data acquisition module 421 and a feedback control module 422. The data acquisition module 421 is installed on the computer 1 and is used to collect the gesture data A, visual data E, and auditory data H generated by the subject during the test. The feedback control module 422 is installed on the tablet computer 5 and is used to feedback information and control the auditory test device 2 to emit a corresponding audio signal.

The data operation unit 43 includes a data operation module 431, a memory 432 and a counter 433 which are electrically connected to the main controller 41. The memory 432 is used to store the data collected by the data acquisition module 421. The counter 433 is used to record the data. The data operation module 431 is used to integrate, analyze and operate the data in the memory 432 to form a database and store it in the memory 432.

The sound automatic control unit 44 includes a DSP 441, one end of which is electrically connected to the main controller 41, and the other end is connected to the hearing test device 2. The DSP 441 can automatically control the sound signal output by the hearing test device 2 according to the database. The DSP 441 of this embodiment is a digital signal processor.

The present invention provides a sound control device based on multi-sensory data with a new structure, which can be connected and used in conjunction with hearing test devices such as computers, tablet computers and audiometers. The working principle is: when the computer controls the hearing test device to send a test audio signal, the subject receives the audio signal through the sound-generating device worn, observes the screen of the tablet computer with his eyes, finds out the text signal corresponding to the heard audio signal and responds by clicking the corresponding text information on the screen to form a response action. The first acquisition device records the subject's gestures and vision and feeds them back to the sound control device. The sound control device integrates and analyzes these data with the hearing data and related data and saves the analysis results.

The master controller 41 of this embodiment may be an MCU control chip.

The hearing test device 2 of this embodiment may be a hearing test device such as an audiometer.

The tablet computer 5 of this embodiment can be a tablet computer or the like that can be used as a controller controlled by the subject. The tablet computer 5 includes a touch screen 51, which is electrically connected to the feedback control module 422 and configured to display text information and provide the subject with a response operation; the feedback control module 422 can control the text information to be displayed at different positions of the touch screen 51 and record the position where the subject clicks the touch screen 51, the number of times the touch screen 51 is clicked, and the time of contact with the touch screen 51; the feedback control module 422 of this embodiment can be a dedicated software embedded in the tablet computer 5.

The text information includes text signals corresponding to the audio signal and other text signals not corresponding to the audio signal;

The main controller 41 can control the text signal to be displayed at different positions of the touch display screen 51 .

Therefore, the subject's visual data is recorded while watching the touch display screen 51, the gesture data is recorded during the operation, and the auditory data is recorded during the test. These data are acquired in real time during the test and will not interfere with the test process. Moreover, these data are generated based on the subject's instinctive reaction, with high accuracy and high reference value.

The touch display screen 51 also displays a loudness level table 511 and a volume increase/decrease button 512. The loudness level table 511 includes four levels: quiet S, medium M, loud L, and noisy N. The loudness step range of the loudness level table 511 is 5~10dB. The volume increase/decrease button 512 is used to control the output sound volume of the hearing test device 2. The loudness level table 511 can be used by the subject to determine the autonomously perceived loudness level and the data is recorded by the counter 433 and stored in the memory 432.

The data acquisition unit 42 of this embodiment can store the collected data in the memory 432, including the text or image information corresponding to the audio signal emitted by the hearing test device. For example: the hearing test device 2 emits the voice of "banana" (the loudness can be controlled at 15dBHL above the threshold), and the touch screen 51 displays the words "banana", "rubber", "beef" and "box", and the touch screen 51 displays a "re-listen" button. After clicking the "re-listen" button, the hearing test device 2 plays the same voice again, but the position of the text corresponding to the voice changes. The subject can click the text corresponding to the voice heard. After clicking correctly, the audiometer will emit a voice with a loudness of 5dBHL above the threshold. The subject also clicks the corresponding text. After confirmation, the voice loudness continues to increase, and the subject is required to adjust the volume increase and decrease button 512 according to the "loudness level table 511" displayed on the screen. Thus, a loudness level table 511 and volume increase/decrease buttons 512 are displayed on the touch display screen 51. As the volume increases, the subject can determine which level of the loudness level table 511 the subject perceives autonomously. After confirmation, the data is recorded by the counter 433 and stored in the memory 432.

The gesture data A includes at least the number of times the touch display screen 51 is clicked and the time interval between clicks of the touch display screen 51; the visual data E includes at least position information, volume increase and decrease buttons 512, loudness level table 511, etc.; the auditory data H includes at least audio data, loudness data, and spectrum data related to text information.

The data calculation module 431 first establishes a multi-sensory database based on the gesture data A, the visual data E, and the auditory data H, and then establishes a data table based on the data in the multi-sensory database. The data table can at least reflect the functional relationship between frequency and gesture, the functional relationship between loudness increment and gesture, and the functional relationship between frequency, loudness and gesture.

The multi-sensory database obtained by the data calculation module 431 is stored in the memory 432 as the basic data for the DSP 441 to automatically control the volume. The basic data of the multi-sensory database at least includes loudness level values of different frequencies, loudness increment response time values of different frequencies, random position change response accuracy, and the difference between the number of clicks and the number of sounds.

The loudness level values of different frequencies include the perception threshold, S value, M value, L value, N value, etc. These thresholds are related to age, hearing loss status, ear habits, physical condition, emotions, etc.

The loudness increment response time values at different frequencies record the response time of volume control, that is, when the stimulus sound volume exceeds the set loudness level, the subject needs to click the volume increase and decrease button 512 to adjust the perceived volume to match the set loudness level, and at this time, the response time interval of clicking the button is recorded;

Random position change response accuracy: Since the order or position of the text signals corresponding to the same audio signal on the touch screen is random, this design facilitates the identification of the correlation between auditory and visual and gesture behaviors;

The difference between the number of clicks and the number of sounds can be used to record the number of times the subject clicks the button and the response time of the click gesture, etc.

The above data can be repeatedly collected and compared with the data of multiple tests for analysis and adjustment to establish the basis for machine learning, and then a mobile phone or other computer 1 can be used to automatically analyze the correlation between individual hearing status and needs.

These data establish a complete "ear-hand-eye EHE" integrated operation method and a method for recording auditory perception status. The data related to clarity is recorded in the position recognition of the speech signal; the data related to sensitivity is recorded in the manual time information after the stimulus sound; and the data related to the volume increment ΔP is recorded in the key press number information of the volume increase and decrease control.

These data form a three-dimensional "gesture-vision-hearing" data set, which is stored in the computer 1 database like a conventional audiogram.

The data acquisition unit 42 records the data related to clarity in the position identification of the speech signal, records the data related to sensitivity in the manual time information after the stimulation sound, and records the data related to the volume increment ΔP in the key press number information of the volume increase and decrease control. These data form a three-dimensional "gesture-vision-hearing" data set, which is stored in the database of the computer 1 like a conventional audiogram.

DSP 441 is embedded with algorithm tables that can process speech signals and music signals of ear sounds separately. The purpose of speech signals is to improve or maintain clarity; the purpose of music signals is to make them pleasant to the ears. The algorithm tables include speech mode algorithm tables and music mode algorithm tables. DSP 441 can automatically switch to speech mode or music mode according to the input signal. Therefore, different modes can be switched according to different sound signals to meet the use requirements.

DSP 441 can automatically configure the dynamic sound pressure range ΔD and the incremental value ΔP based on the data in the multi-sensory database and the algorithm table, and automatically configure the frequency band range ΔF according to the input signal. The frequency band range ΔF, the dynamic sound pressure range ΔD, the incremental value ΔP and other data will be stored in the RAM memory of DSP 441 in the form of data packets. The format of each incremental item in the data packet is stored in a manner not less than n orders, and the n value can be modified and expanded; the control parameters set for the voice mode and the music mode are different, and the control parameters are configured in the form of data packets. In this way, a larger hearing threshold range can be covered and the applicability is enhanced. More importantly, the data after the in-ear sound signal is adjusted will be stored in the main controller. The main controller 41 is a microchip, which can be directly installed in products such as hearing aids or headphones. It can be used directly after assembly, and is suitable for individuals with different hearing conditions, effectively making up for the shortcomings of existing headphones or hearing aids in sound control.

As shown in FIGS. 2 and 4 , the sound control device 4 of this embodiment further includes a communication module 45, a PCB board 46 and a plurality of ports 47 for connection. The host 40 of this embodiment can be an independent housing, the master controller 41, DSP, etc. are installed on the PCB board, and the port 47 is opened on the surface of the host. The sound control device 4 can communicate with the computer 1 through the communication module 45 and upload the above data to the computer 1. It is convenient to check the stored information installed in the computer 1 later, and analyze and integrate the correlation between hearing, vision and gesture.

As shown in FIGS. 5 and 6 , the sound control device based on multi-sensory data includes at least:

Computer 1;

A hearing test device 2, connected to the computer 1, configured to generate an audio signal;

a sound-generating device 3 connected to the hearing test device 2 and configured to transmit an audio signal to the ear of the subject;

The sound control device 4 is connected to the computer 1 and the hearing test device 2 and is configured to generate a text signal according to the audio signal;

Tablet computer 5, connected to the sound control device 4, configured to display text information and allow the subject to respond and operate;

A first collection device 6, connected to the sound control device 4, configured to collect the subject's actions and vision during the response process;

The first acquisition device 6 of this embodiment can be a camera, and the number and position can be set according to specific needs, for example: set on the tablet computer 5, set on the outer shell of the sound control device 4 (as shown in FIG. 4 ).

The sound control device based on multi-sensory data also includes a second acquisition device 7, which is electrically connected to the main controller 41 and is configured to collect the voice emitted by the subject during the test. Therefore, some subjects will make sounds during the test, and the second acquisition device 7 can collect the sounds emitted by the subjects and feed them back to the main controller 41, which will analyze the data with other relevant data. The second acquisition device 7 of this embodiment can be a microphone, which can be specifically set on the tablet computer 5, or on the sound device 3, or on the host 40 of the sound control device 4 (as shown in Figure 4), pointing to the mouth of the subject.

The main working principle of the sound control device based on multi-sensory data of the present invention is carried out through the following steps:

S1. Connect all devices and turn on the machine;

S2, the subject wears a sound-generating device 3 such as headphones, and the doctor or staff operates the computer 1 to control the hearing test device 2 to emit an audio signal;

S3, the subject observes the touch screen 51 of the tablet computer 5 with his eyes, finds out the text signal corresponding to the heard audio signal and responds by clicking the corresponding text information on the touch screen 51;

S4, when performing step S3, the first acquisition device 6 works to film or take a photo of the subject's response gesture and feed it back to the main controller 41, the second acquisition device 7 works to pick up the voice signal emitted by the subject during the test and feed it back to the main controller 41, and the feedback control module 422 feeds back the relevant information of the subject clicking the touch display screen 51 to the main controller 41;

S5. The feedback control module 422 of the data acquisition unit 42 cooperates with the data acquisition module 421 to record and store various data such as gesture data A, visual data E, and auditory data H generated during the test in the memory 432;

S6, the data operation module 431 of the data operation unit 43 first establishes and stores a multi-sensory database according to the gesture data A, the visual data E, and the auditory data H in the memory 432, and then establishes a data table according to the data in the multi-sensory database;

S7, the DSP 441 of the sound automatic control unit 44 automatically controls the sound signal emitted by the hearing test device 2 based on the data table, data packet and/or various basic data in the database using the algorithm table embedded in the DSP 441, thereby realizing the control of the in-ear sound signal;

S8. The sound control device 4 can communicate with the computer 1 and upload the data to the computer 1.

The present invention provides a sound control device based on multi-sensory data with a new structure. The working principle of the device is as follows: when a computer 1 controls a hearing test device 2 to emit a test audio signal, the subject receives the audio signal through a worn sound-generating device 3, observes the screen of a tablet computer 5 with his eyes, finds out the text signal corresponding to the heard audio signal, and responds by clicking on the corresponding text information on the screen to form a response action. A first acquisition device 6 records the subject's gestures and vision and feeds them back to a sound control device 4. The sound control device 4 integrates and analyzes these data with the hearing data and related data and saves the analysis results.

The present invention designs a system that can collect and record a person's multi-sensory data in real time. The system can at least collect and record gesture, vision, hearing and other data and related data at the same time, and use a signal to stimulate and obtain gesture, vision and hearing reaction data at the same time, and integrate and analyze these data to judge the subject's consciousness or demand for signals. The multi-sensory data finally obtained has strong reference value and can be applied to skill training, language learning, recognition ability training, machine learning and other fields, and has a wide range of applications.

More importantly, the device can also perform calculations based on multi-sensory data to form a three-dimensional "gesture-vision-hearing" database, and the DSP 441 can automatically adjust the sound signal output by the hearing test device 2 according to the data in the database.

The present invention uses a special hardware architecture to establish a digitization process for multi-sensory input signal response data, reintegrates the data ignored by traditional hearing tests into a three-dimensional data set, and finely regulates the sound input to the human ear based on the data set, while meeting the multiple needs of listening to speech and appreciating music.

Compared with the traditional hearing threshold data format, which is limited to the annotation of hearing status, the present invention can establish a new hearing data set, which can meet the needs of scientific research on auditory perception characteristics, prevention of auditory diseases, and detection of changes in brain cognitive status.

More importantly, the data after the in-ear sound signal is adjusted will be stored in the main controller, which is a microchip that can be directly installed in products such as hearing aids or headphones. After assembly, it can be used directly and is suitable for individuals with different hearing conditions, effectively making up for the shortcomings of existing headphones or hearing aids in sound control.

The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims (6)

1. A sound control device based on multi-sensory data, connected to a computer (1), a hearing test device (2), and a tablet computer (5), characterized in that the sound control device comprises a master controller (41), a data acquisition unit (42) connected to the master controller (41), a data operation unit (43), and a sound automatic control unit (44); the data acquisition unit (42) comprises a data acquisition module (421) and a feedback control module (422), the data acquisition module (421) being installed on the computer (1) and being used to collect gesture data A, visual data E, and hearing data H generated by a subject during a hearing test; the feedback control module (422) being used to feedback information and control the hearing test device (2) to emit a corresponding audio signal; the data operation unit (43) being used to control the hearing test device (2) to emit a corresponding audio signal; the data operation unit (42) being used to control the hearing test device (2) to emit a corresponding audio signal; the data operation unit (43 ... (43) comprises a data operation module (431), a memory (432) and a counter (433), wherein the memory (432) is used to store the data collected by the data collection module (421), the counter (433) is used to record the data, and the data operation module (431) is used to integrate, analyze and operate the data in the memory (432) to form a multi-sensory database and store it in the memory (432); the automatic sound control unit (44) comprises a DSP (441), one end of the DSP (441) is electrically connected to the main controller (41), and the other end is connected to the hearing test device (2), and the DSP (441) can automatically control the sound signal output by the hearing test device (2) according to the multi-sensory database; The tablet computer (5) comprises a touch screen (51), the touch screen (51) being electrically connected to the feedback control module (422) and configured to display text information and provide a subject with a means of responding and operating; the feedback control module (422) is capable of controlling the text information to be displayed at different positions of the touch screen (51) and recording the position at which the subject clicks the touch screen (51), the number of times the subject clicks the touch screen (51), and the time the subject is in contact with the touch screen (51); The text information includes a text signal corresponding to the audio signal emitted by the hearing test device (2) and other text signals not corresponding to the audio signal; The master controller (41) is capable of controlling the text signal to be displayed at different positions of the touch display screen (51); The touch display screen (51) also displays a loudness level meter (511) and a volume increase/decrease button (512); the loudness level meter (511) includes four levels: soft sound S, medium sound M, loud sound L, and noisy sound N; the loudness step range of the loudness level meter (511) is 5-10 dB; the volume increase/decrease button (512) is used to control the output sound volume of the hearing test device (2); the loudness level meter (511) is used by the subject to determine the autonomously perceived loudness level and the data is recorded by the counter (433) and stored in the memory (432); The gesture data A at least includes the number of times the touch display screen (51) is clicked and the time interval between clicks on the touch display screen (51); the visual data E at least includes position information, volume increase and decrease buttons (512), and a loudness level table (511); the auditory data H at least includes audio data, loudness data, and spectrum data related to the text information; The data operation module (431) first establishes a multi-sensory database based on the gesture data A, the visual data E, and the auditory data H, and then establishes a data table based on the data in the multi-sensory database, wherein the data table at least reflects the functional relationship between frequency and gesture, the functional relationship between loudness increment and gesture, and the functional relationship between frequency, loudness, and gesture. 2. The sound control device based on multi-sensory data according to claim 1 is characterized in that the multi-sensory database calculated by the data calculation module (431) is stored in the memory (432) as basic data for the DSP (441) to automatically control the volume, and the basic data of the multi-sensory database at least includes loudness level values of different frequencies, loudness increment response time values of different frequencies, random position change response accuracy, and the difference between the number of clicks and the number of sounds. 3. The sound control device based on multi-sensory data according to claim 2 is characterized in that the DSP (441) is embedded with at least an algorithm table for processing the speech signal and the music signal of the ear sound respectively; the algorithm table includes at least two types: a speech mode algorithm table and a music mode algorithm table; the DSP (441) can automatically switch to the speech mode or the music mode according to the input signal. 4. The sound control device based on multi-sensory data according to claim 3 is characterized in that the DSP (441) can automatically configure the dynamic sound pressure range ΔD and the incremental value ΔP based on the data in the multi-sensory database in conjunction with the algorithm table, and automatically configure the frequency band range ΔF according to the input signal, and store the dynamic sound pressure range ΔD, the incremental value ΔP, and the frequency band range ΔF in the form of a data packet in the built-in RAM; the format of each incremental item in the data packet is stored in a manner not less than n orders, and the value of n can be modified and expanded; the control parameters set for the voice mode and the music mode are different, and the control parameters are configured using the data packet. 5. A sound control device based on multi-sensory data, characterized in that it at least includes: Computer (1); A hearing test device (2), connected to the computer (1) and configured to generate an audio signal; A sound-generating device (3), connected to the hearing test device (2), and configured to transmit the audio signal to the ear of the subject; A sound control device (4), connected to the computer (1) and the hearing test device (2), and configured to generate a text signal according to the audio signal; A tablet computer (5), connected to the sound control device (4), configured to display the text information and allow the subject to respond and operate; A first collection device (6), connected to the sound control device (4), configured to collect the subject's actions and vision during the response process; The sound control device (4) is the sound control device according to any one of claims 1 to 4; The data acquisition module (421) is installed on the computer (1). 6. The sound control device based on multi-sensory data according to claim 5 is characterized in that it also includes a second collection device (7), which is electrically connected to the main controller (41) of the sound control device (4) and is configured to collect the voice emitted by the subject during the test.