CN107248407A - A kind of gesture musical performance device - Google Patents

Abstract

The invention discloses a gesture music player, which comprises a shell and a power supply arranged in the shell, an integrated circuit, a memory, an image acquisition device, a display, a central processing unit, an audio output device and a gesture classifier, the image acquisition device and the gesture classifier. The central processing unit is connected by a data bus; the gesture classifier is connected to the central processing unit, and standard gestures are pre-stored in the gesture classifier, and the central processing unit preprocesses the image information collected by the image acquisition device, and communicates with the The standard gestures in the gesture classifier are compared, and the recognition result is obtained according to the comparison information. The present invention can be realized only by using a device with a camera and an audio output device, and is simple and easy to implement. The user controls the device to emit different sounds through different gestures to realize the performance of the music. By defining specific gestures, the performance function of virtual musical instruments can be realized, and the fun of music creation can be enjoyed anytime and anywhere.

Description

A gesture music player

technical field

The invention relates to a gesture music player, in particular to the fields of gesture recognition and music performance, in particular to a system for playing different musical notes according to different recognized gestures.

Background technique

In the existing technology, there are applications that can control the playback and pause of music and video, and adjust the volume through gestures. There are also software that can interact with the outside world through gestures through wearable devices such as VR glasses, VR helmets and other special virtual devices. Users need to spend extra money to buy these devices when they experience it. It is inconvenient to carry, and the use is more complicated, and it is necessary to connect the peripheral device with the corresponding computer.

Currently in the field of gesture recognition, especially in the field of gesture recognition of mobile smart terminals. There is still no system that can perform and create music only through gesture changes.

In view of the above problems, the present invention provides a gesture music player, aiming to develop a device that can control the music performance of the device only by using the camera of the smart device, and play notes of different pitches through changes in user gestures. Realize the performance of the music. Increase the direct interaction between people and equipment during music creation and playback, adding fun to music.

Contents of the invention

In order to achieve the above object, the present invention provides the following technical solutions: a gesture music player, which includes a housing and a power supply, an integrated circuit, a memory, an image acquisition device, a display, a central processing unit, an audio output device and a gesture device arranged in the housing. The classifier is characterized in that the power supply increases the required voltage for each component; the image acquisition device is connected to the central processing unit using a data bus, and the image acquisition device is responsible for using the image information of the gesture, and the image The information is transmitted to the central processing unit for processing;

The display is connected to the central processing unit by a data bus, so as to display the picture collected by the image acquisition device;

The gesture classifier is connected to the central processing unit, and standard gestures are pre-stored in the gesture classifier;

The central processing unit preprocesses the image information collected by the image collection device, compares it with the standard gesture in the gesture classifier, and obtains a recognition result according to the comparison information;

The audio output device is directly connected to the central processing unit by means of a data bus, and when the central processing unit obtains the recognition result, it selects the corresponding audio file from the database and transfers it to it and controls it to play.

Further, preferably, the image acquisition device communicates with the data bus using a MIPI interface, the audio output device connects with the data bus using a D/A interface, and the display communicates with the data bus through a man-machine interface. communication.

Further, preferably, the image acquisition device adopts a camera, and the audio output device adopts a speaker.

Further, as a preference, the method for the central processor to preprocess the image information collected by the image collection device is: convert the incoming and collected image into a grayscale image and perform gesture segmentation and feature extraction to obtain the gesture area overall profile information.

Further, as a preference, the comparison method when comparing with the standard gesture in the gesture classifier is as follows: the classifier uses a distance matching algorithm to match and recognize the input contour information with the gestures existing in the classifier, and send the recognition result to The cpu, cpu and memory unit record and save the currently played music.

Further, preferably, the musical instrument is provided with a small window for the camera to display the user's real-time gestures, a parameter selection button for the user to provide parameters necessary for playing music, a playback display interface, a save button, a pause button, and a return and exit button.

Further, the present invention provides a kind of performance method of gesture music player, it is characterized in that: it comprises the following steps:

(1) Turn on the power of the gesture music player and click to enter the main interface, select the playback parameters: tone, timbre and eight beats;

(2) After selecting and clicking OK, the camera starts to work, starts to monitor the gestures in the screen, and transmits the screen to the display screen in real time, and the gesture classifier starts to receive gesture information;

(3) The gesture classifier compares each frame of gestures with the gestures in the pre-trained gesture classifier. If they match, a monitoring event is triggered, and the monitoring event is processed, and the matching file in the music file database is output through the speaker. , complete the vocalization of the musical symbol represented by this gesture;

(4) Repeat step (3) until a stop gesture is encountered or the user clicks the pause button;

(5) The user clicks Save to generate the current performance audio file and save it locally.

Compared with prior art, the beneficial effect of the present invention is:

The present invention can be realized only by using a device with a camera and an audio output device, and is simple and easy to implement. The user controls the device to emit different sounds through different gestures to realize the performance of the music. By defining specific gestures, the performance function of virtual musical instruments can be realized, and the fun of music creation can be enjoyed anytime and anywhere.

Description of drawings

Fig. 1 is a block diagram of a kind of gesture music player of the present invention;

Fig. 2 is a hardware connection diagram of a kind of gesture music player of the present invention;

Fig. 3 is the performance flowchart of a kind of gesture music player of the present invention;

Fig. 4 is the plane structure diagram of a kind of gesture music player of the present invention;

Among them, 1. Image acquisition device, 2. Display, 3. Audio output device, 4. CPU, 5. Gesture classifier, 6. Camera, 7. Speaker is system input and output device, 8. Touch screen, 9. MIPI interface , 10, D/A interface, 11, man-machine interface, 12, RAM, 14, ROM, 15, small window, 16, parameter selection button. 17. Play display interface, 18. Save button, 19. Pause button, 20. Return and exit button.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to Fig. 1-4, the present invention provides a kind of technical scheme: a kind of gesture music player, it comprises shell and the power supply that is arranged in the shell, integrated circuit, memory, image acquisition equipment 1, display 2, central processing unit CPU4 . Audio output device 3 and gesture classifier 5, characterized in that the power supply increases the required voltage for each component; the image acquisition device 1 is connected to the central processing unit 4 using a data bus, and the image acquisition device Responsible for adopting gesture image information, and transmitting the image information to the central processing unit for processing. The main working process of the image acquisition device is: capturing user gesture image information, digitizing the image information and then transferring it to the processor;

The display 2 is connected to the central processing unit 4 by a data bus, so as to display the pictures collected by the image acquisition device. The display is a touch screen, which is the entrance for user operation. Its working process is: the user's operation information is transmitted to the processor, and after processing, the processing result of the processor is returned. When the image acquisition device is turned on, the picture of the image acquisition device is displayed in real time, which is convenient for the user to make adjustments; the gesture classifier 5 is connected to the central processing unit 4, and standard gestures are pre-stored in the gesture classifier 5; The central processing unit 4 preprocesses the image information collected by the image acquisition device 1, and compares it with the standard gesture in the gesture classifier 5, and obtains a recognition result according to the comparison information;

The audio output device 3 is directly connected to the central processing unit by means of a data bus. When the central processing unit obtains the recognition result, it selects the corresponding audio file from the database and transfers it to it and controls it to play.

Among them, the cpu and the gesture classifier are the two most important devices in the present invention. The cpu preprocesses the data imported by the image acquisition device, converts the incoming image into a grayscale image and performs gesture segmentation and feature extraction to obtain the gesture area overall profile information. The classifier uses the distance matching algorithm to match and recognize the input contour information with the gestures existing in the classifier, and sends the recognition results to the CPU. The cpu and the memory unit can also record and save the currently played music.

The speaker is the audio output device and is connected directly to the processor. When the cpu gets the recognition result, select the corresponding audio file from the database and pass it in and control it to play

In this embodiment, the image acquisition device 1 uses the MIPI interface 9 to communicate with the data bus, the audio output device uses the D/A interface 10 to connect to the data bus, and the display is connected to the data bus through the man-machine interface 11 communicate with the data bus.

Wherein, the image acquisition device adopts a camera, and the audio output device adopts a speaker. The method for the central processor to preprocess the image information collected by the image collection device is: convert the incoming and collected image into a grayscale image and perform gesture segmentation and feature extraction to obtain the overall contour information of the gesture area.

In addition, the comparison method when comparing with the standard gesture in the gesture classifier is: the classifier uses a distance matching algorithm to match and recognize the input contour information with the gestures in the classifier, and send the recognition result to the cpu, cpu and the memory unit to record and save the currently played music.

For ease of operation, the instrument is provided with a small window 15 for displaying user real-time gestures by a camera, a parameter selection button 16 for the user to provide parameters necessary for playing music, a playback display interface 17, a save button 18, a pause button 19, and a return button. Exit key 20.

In the present embodiment, the present invention also provides a method for playing a gesture music player, which is characterized in that: it comprises the following steps:

(1) Turn on the power of the gesture music player and click to enter the main interface, select the playback parameters: pitch, timbre and eight beats, the main interface can select the sound playback parameters: select the timbre of the existing musical instruments in reality (such as piano , trumpet, violin and other common instruments), music tune A~G and music beat 1/4, 3/4, etc.;

(2) After selecting and clicking OK, the camera starts to work, starts to monitor the gestures in the screen, and transmits the screen to the display screen in real time, and the gesture classifier starts to receive gesture information;

(3) The gesture classifier compares each frame of gestures with the gestures in the pre-trained gesture classifier. If they match, a monitoring event is triggered, and the monitoring event is processed, and the matching file in the music file database is output through the speaker. , complete the vocalization of the musical symbol represented by this gesture;

(4) Repeat step (3) until a stop gesture is encountered or the user clicks the pause button;

(5) The user clicks Save to generate the current performance audio file and save it locally.

Among them, between steps (1) and (2), there are: after a series of selections by the user, the system finds the corresponding sound file database to be played according to the user's selection. If the database is too large, it can be considered to be placed on the server side. , obtained in real time when used.

Among them, the key technology between step (2) and step (3) is the generation of gesture classifiers. The present invention adopts the method of machine learning, by opencv or other computer vision software, uses known machine learning algorithm to generate gesture recognition classifier through the training of a large number of positive and negative samples. In particular, before the classifier performs classification in step 3, certain feature extraction and processing should be performed on the read frame data to improve the success rate and efficiency of recognition. Detailed description will be given below.

The image acquisition device (camera), responsible for the acquisition and transmission of gesture information, is connected to the CPU through an interface. Its main working process is: capturing user gesture image information, digitizing the image information and transferring it to the processor.

The display is the touch screen, which is the entrance for user operation. Its working process is: the user's operation information is transmitted to the processor, and after processing, the processing result of the processor is returned. When the image acquisition device is turned on, the image of the image acquisition device is displayed in real time, which is convenient for the user to make adjustments. The cpu and gesture classifier are the two most important devices in the present invention. The cpu preprocesses the incoming data of the image acquisition device, converts the incoming image into a grayscale image and performs gesture segmentation and feature extraction to obtain the whole of the gesture area. profile information. The classifier uses the distance matching algorithm to match and recognize the input contour information with the gestures existing in the classifier, and sends the recognition results to the CPU. The cpu and the memory unit can also record and save the currently played music. The speaker is the audio output device and is connected directly to the processor. When the cpu gets the recognition result, it selects the corresponding audio file from the database and passes it in and controls it to play.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (7)

1. a kind of gesture musical performance device, it includes shell and the power supply being arranged in shell, integrated circuit, memory, image Collecting device, display, central processing unit, audio output apparatus and gesture classifier, it is characterised in that the power supply is each Voltage needed for part is improved；Described image collecting device is connected with the central processing unit using data/address bus, and described image Collecting device is responsible for the image information using gesture, and image information is transferred into the central processing unit handled；

The display is connected with the central processing unit using data/address bus, the picture gathered so as to display image collecting device Face；

The gesture classifier is connected with the central processing unit, and is prestored in the gesture classifier provided with standard gesture；

The central processing unit is pre-processed to described image collecting device acquired image information, and is divided with the gesture Standard gesture in class device is contrasted, and is identified result according to comparative information；

The audio output apparatus is directly connected with central processing unit using data/address bus, when central processing unit is identified result When, select corresponding audio file is incoming wherein and to control it to play out from database.

2. a kind of gesture musical performance device according to claim 1, it is characterised in that：Described image collecting device is used MIPI interfaces and the data/address bus connection communication, the audio output apparatus are connected using D/A interfaces with the data/address bus, The display passes through man-machine interface and the data bus communication.

3. a kind of gesture musical performance device according to claim 1, it is characterised in that：Described image collecting device is used and taken the photograph As head, the audio output apparatus uses loudspeaker.

4. a kind of gesture musical performance device according to claim 1, it is characterised in that：The central processing unit is to the figure As the method that collecting device acquired image information is pre-processed is：The image of incoming collection is switched into gray level image to go forward side by side Row Hand Gesture Segmentation and feature extraction, obtain the overall profile information of gesture area.

5. a kind of gesture musical performance device according to claim 1, it is characterised in that：With the mark in the gesture classifier Control methods when quasi- gesture is contrasted is：Grader is used will input profile information with being deposited in grader apart from matching algorithm Gesture carry out match cognization, and recognition result is sent into cpu, cpu and internal storage location are remembered to the music currently played Record is with preserving.

6. a kind of gesture musical performance device according to claim 1, it is characterised in that：Camera is provided with the performer The wicket of user's real-time gesture is shown, the parameter selection key of parameter selection necessary to performing music is provided the user, broadcasts Display interface is put, button, pause button, return is preserved and exits button.

7. a kind of playing method of gesture musical performance device, it is characterised in that：It comprises the following steps：

（1）Open the power supply of gesture musical performance device and click to enter main interface, select play parameter：Tone, tone color and eight clap；

（2）Clicked on after selection after determining, camera is started working, start to monitor the gesture in picture, and picture is transmitted in real time Onto display screen, while gesture classifier starts to receive gesture information；

（3）Gesture classifier is contrasted the gesture in each frame gesture gesture classifier good with training in advance, if matching Monitoring event is then triggered, monitoring event is handled it, the file matched in music file database is exported by loudspeaker, Complete the sounding of happy symbol representated by this gesture；

（4）Repeat step（3）, until running into stopping gesture or user's click pause button；

（5）User, which clicks on, to be preserved, and generation is current to play audio file and be saved in local；

Wherein, the gesture musical performance device is the gesture musical performance device described in claim 1-6 any one.