CN106601217B - A kind of interactive musical instrument performance method and device - Google Patents
A kind of interactive musical instrument performance method and device Download PDFInfo
- Publication number
- CN106601217B CN106601217B CN201611108803.4A CN201611108803A CN106601217B CN 106601217 B CN106601217 B CN 106601217B CN 201611108803 A CN201611108803 A CN 201611108803A CN 106601217 B CN106601217 B CN 106601217B
- Authority
- CN
- China
- Prior art keywords
- target
- pattern
- musical instrument
- action
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000002452 interceptive effect Effects 0.000 title claims abstract description 26
- 230000009471 action Effects 0.000 claims abstract description 142
- 230000003068 static effect Effects 0.000 claims abstract description 46
- 230000033001 locomotion Effects 0.000 claims description 80
- 230000001133 acceleration Effects 0.000 claims description 15
- 230000036544 posture Effects 0.000 claims 19
- 230000008859 change Effects 0.000 description 9
- 210000000988 bone and bone Anatomy 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000003993 interaction Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 4
- 241001465754 Metazoa Species 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000010079 rubber tapping Methods 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 241000283973 Oryctolagus cuniculus Species 0.000 description 1
- 241000282320 Panthera leo Species 0.000 description 1
- 241000282376 Panthera tigris Species 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/0008—Associated control or indicating means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- General Physics & Mathematics (AREA)
- Electrophonic Musical Instruments (AREA)
Abstract
本发明实施例提供了一种交互式乐器演奏方法及装置，方法包括：获得预设时长内目标用户的预设数量个第一数据；根据各第一数据，确定目标用户的动作类别，其中，动作类别包括静态姿势或动态动作；当目标用户的动作类别为静态姿势时，根据每个第一数据确定目标用户的目标姿势，并根据目标姿势以及预设的乐器种类模板库，确定目标姿势对应的目标乐器种类；根据目标乐器种类以及预设图案库，显示与目标乐器种类对应的图案；当目标用户的动作类别为动态动作时，根据每个第一数据确定目标用户的目标动作，并根据目标动作、当前显示图案以及预设音频库，输出与当前显示图案对应的音频。应用本发明实施例，实现了交互式乐器演奏。
Embodiments of the present invention provide a method and device for playing an interactive musical instrument. The method includes: obtaining a preset number of first data of a target user within a preset duration; and determining an action category of the target user according to each first data, wherein, The action category includes static posture or dynamic action; when the action category of the target user is a static posture, the target posture of the target user is determined according to each first data, and the corresponding target posture is determined according to the target posture and the preset template library of musical instrument types. According to the target musical instrument type and the preset pattern library, the pattern corresponding to the target musical instrument type is displayed; when the action category of the target user is a dynamic action, the target action of the target user is determined according to each first data, and according to The target action, the current display pattern and the preset audio library, output the audio corresponding to the current display pattern. By applying the embodiments of the present invention, interactive musical instrument performance is realized.
Description
Technical Field
The invention relates to the technical field of man-machine interaction, in particular to a playing method and device of an interactive musical instrument.
Background
The man-machine interaction technology is a technology for realizing man-computer interaction in an effective mode through computer input and output equipment. With the development of human-computer interaction technology, the interface between human and computer is changing constantly. The user interface is developed from the most original command line user interface (text command input is carried out through a keyboard, and text information output is carried out through a display) to the graphical user interface (text is input through the keyboard, a cursor is controlled through a mouse, and graphs are displayed through the display), and the interaction efficiency of a user and a computer is greatly improved due to the innovation of the user interface. With the invention of updating hardware and the design of computer algorithm, three-dimensional motion capture equipment such as a depth camera and the like are widely applied, and a natural user interface is generated and developed rapidly. The natural user interface captures motion posture data of a human body through the three-dimensional motion capture equipment and sends the motion posture data to the computer, the computer analyzes and processes the motion posture data of the human body, for example, motions of waving hands, jumping and the like of a user are recognized, and the recognized motions are further converted into commands, so that human-computer interaction is achieved.
The interactive musical instrument performance refers to a process of performing motion analysis and musical performance analysis by a computer by capturing human body motion, and then performing pattern display and audio output. At present, a natural user interface captures motion posture data of a human body by using a three-dimensional motion capture device, and a technology for realizing interactive musical instrument playing by capturing the motion posture data of the human body by using the three-dimensional motion capture device is lacked. Therefore, a technique for realizing an interactive musical instrument performance by capturing human body motions with a three-dimensional capturing apparatus is required.
Disclosure of Invention
The embodiment of the invention aims to provide an interactive musical instrument playing method and device so as to realize interactive musical instrument playing.
In order to achieve the above object, the embodiment of the present invention discloses a method for playing an interactive musical instrument, the method comprising:
acquiring a preset number of first data of a target user within a preset time length;
determining an action type of the target user according to the first data, wherein the action type comprises a static gesture or a dynamic action;
when the action type of the target user is a static posture, determining the target posture of the target user according to each first datum, and determining a target musical instrument type corresponding to the target posture according to the target posture and a preset musical instrument type template library; displaying a pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library;
and when the action category of the target user is a dynamic action, determining the target action of the target user according to each first datum, and outputting the audio corresponding to the current display pattern according to the target action, the current display pattern and a preset audio library.
Optionally, the determining the action category of the target user according to each piece of first data includes:
judging whether the first data meet similar conditions or not;
if yes, determining the action category of the target user to be a static gesture;
and if not, determining the action category of the target user to be a dynamic action.
Optionally, the step of storing a corresponding relationship between each instrument type and a static posture in the instrument type template library, and determining a target instrument type corresponding to the target posture according to the target posture and a preset instrument type template library includes:
searching a static posture which is the same as the target posture in the preset musical instrument type template library;
and determining the musical instrument type corresponding to the searched static posture according to the musical instrument type template library, and determining the musical instrument type as the target musical instrument type corresponding to the target posture.
Optionally, the step of displaying the pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library includes:
determining a target pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library;
judging whether a displayed pattern exists at present;
if no pattern is currently displayed, displaying the target pattern;
if the pattern displayed currently exists, judging whether the target pattern is consistent with the pattern displayed currently; if yes, keeping the currently displayed pattern; and if not, displaying the target pattern.
Optionally, the step of outputting an audio corresponding to the current display pattern according to the target action, the current display pattern, and a preset audio library includes:
determining a target pattern corresponding to the target action according to a corresponding relation between a preset pattern and the action;
judging whether the target pattern is consistent with the currently displayed pattern;
and if so, outputting the audio corresponding to the currently displayed pattern according to the preset audio library.
The embodiment of the invention also discloses an interactive musical instrument playing device, which comprises:
the device comprises an obtaining module, a processing module and a processing module, wherein the obtaining module is used for obtaining a preset number of first data of a target user within a preset time length;
the determining module is used for determining the action type of the target user according to the first data, wherein the action type comprises a static gesture or a dynamic action;
the display module is used for determining the target posture of the target user according to each first datum when the determining module determines that the action type of the target user is a static posture, and determining the target musical instrument type corresponding to the target posture according to the target posture and a preset musical instrument type template library; displaying a pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library;
and the output module is used for determining the target action of the target user according to each first datum when the action type of the target user is determined to be the dynamic action by the determination module, and outputting the audio corresponding to the current display pattern according to the target action, the current display pattern and a preset audio library.
Optionally, the determining module includes:
the first judgment submodule is used for judging whether the first data meet the similar conditions or not;
the first determining submodule is used for determining that the action type of the target user is a static posture when the judgment result of the first judging submodule is yes;
and the second determining submodule is used for determining the action type of the target user as a dynamic action when the judgment result of the first judging submodule is negative.
Optionally, the musical instrument category template library stores a corresponding relationship between each musical instrument category and a static posture, and the display module includes:
the searching sub-module is used for searching a static posture which is the same as the target posture in the preset musical instrument type template library;
and the third determining submodule is used for determining the musical instrument type corresponding to the searched static posture according to the musical instrument type template library and determining the musical instrument type as the target musical instrument type corresponding to the target posture.
Optionally, the display module includes:
a fourth determining submodule, configured to determine, according to the target instrument type and a preset pattern library, a target pattern corresponding to the target instrument type;
the second judgment submodule is used for judging whether a display pattern exists currently;
the first display sub-module is used for displaying the target pattern when the judgment result of the second judgment sub-module is negative;
the second display submodule is used for judging whether the target pattern is consistent with the currently displayed pattern or not when the judgment result of the second judgment submodule is yes; if yes, maintaining the currently displayed pattern; and if not, displaying the target pattern.
Optionally, the output module includes:
the fifth determining submodule is used for determining a target pattern corresponding to the target action according to the corresponding relation between the preset pattern and the action;
a third judging submodule for judging whether the target pattern is consistent with the currently displayed pattern;
and the output sub-module is used for outputting the audio corresponding to the currently displayed pattern according to the preset audio library when the judgment result of the third judgment sub-module is yes.
According to the technical scheme, the interactive musical instrument playing method and the device can obtain the preset number of first data of the target user within the preset time length; determining an action type of the target user according to the first data, wherein the action type comprises a static gesture or a dynamic action; when the action type of the target user is a static posture, determining the target posture of the target user according to each first datum, and determining the target musical instrument type corresponding to the target posture according to the target posture and a preset musical instrument type template library; displaying a pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library; and when the action type of the target user is dynamic action, determining the target action of the target user according to each first datum, and outputting the audio corresponding to the current display pattern according to the target action, the current display pattern and the preset audio library, thereby realizing the playing of the interactive musical instrument.
Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flow chart illustrating a method for playing an interactive musical instrument according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an interactive musical instrument playing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention discloses an interactive musical instrument playing method and device, which are respectively explained in detail below.
Referring to fig. 1, fig. 1 is a schematic flow chart of a playing method of an interactive musical instrument according to an embodiment of the present invention, including the following steps:
s101, obtaining a preset number of first data of a target user within a preset time length.
The method provided by the embodiment of the invention can be applied to electronic equipment, for example: and (4) a computer. The embodiment of the present invention is described by taking a computer as an example, and in practical applications, a user may select a type of an electronic device according to a requirement, which is not limited in the present invention.
It should be noted that, in the embodiment of the present invention, the gesture and the motion of the target user may be captured in a three-dimensional space by using a three-dimensional motion capture device externally connected to the computer, so that the computer may obtain the first data of the target user. The three-dimensional motion capture device may be a depth camera (e.g., Kinect), an inertial motion capture device (e.g., gyroscope, accelerometer, etc.), a deformation sensor (e.g., data fabric), etc., which is not limited in this respect. The computer and the three-dimensional motion capture device may adopt a wired connection mode or a wireless connection mode, and the specific connection between the computer and the three-dimensional motion capture device belongs to the prior art, and the details are not repeated herein.
The first data may include at least a three-dimensional position, a three-dimensional motion velocity, a three-dimensional motion acceleration, etc. of the target user's limb. Specifically, a process of capturing a gesture and a motion of a target user in a three-dimensional space by using a three-dimensional motion capture device to obtain first data of the target user belongs to the prior art, and is not described herein again in the embodiments of the present invention.
For example, the preset time period may be 3 seconds, and the three-dimensional motion capture device may collect 30 first data per second, so that 90 first data of the target user may be obtained within the preset time period.
And S102, determining the action type of the target user according to the first data.
Specifically, the motion category includes static gestures or dynamic motions. The computer determines the action category of the target user according to each piece of first data, and the action category may be: judging whether each first data meets a similar condition; if so, determining the action type of the target user as a static gesture; and if not, determining the action category of the target user as the dynamic action.
It should be noted that the step of judging, by the computer, whether each first data satisfies the similar condition may be: when the first data are the same, judging that the first data meet the similar condition; and when at least one first data is different from other at least one first data, judging that the first data do not meet the similar condition.
When the first data includes a plurality of items of content, such as the three-dimensional position, the three-dimensional movement velocity, the three-dimensional movement acceleration, and the like of the limb of the target user, the corresponding content of each first data may be sequentially compared to determine the action category of the target user.
Illustratively, when the number of the obtained first data is 2, the three-dimensional position, the three-dimensional motion velocity, and the three-dimensional motion acceleration of the first data are respectively: 21cm, 5cm, 7cm, 0cm/s, 0m/s2、0m/s2、0m/s2The three-dimensional position, the three-dimensional motion velocity and the three-dimensional motion acceleration of the second first data are respectively as follows: 21cm, 5cm, 7cm, 0cm/s, 0m/s2、0m/s2、0m/s2If the three-dimensional position, the three-dimensional motion speed and the three-dimensional motion acceleration of the first data are respectively judged to be the same as the three-dimensional position, the three-dimensional motion speed and the three-dimensional motion acceleration of the second first data, the first data are judged to meet the similar conditions, and the action type of the target user is determined to be the static posture.
Illustratively, the obtained first data are 2, and the three-dimensional position, the three-dimensional motion velocity and the three-dimensional motion acceleration of the first data are respectively: 21cm, 5cm, 7cm, 0cm/s, 0m/s2、0m/s2、0m/s2Of 1 atThe three-dimensional position, the three-dimensional motion speed and the three-dimensional motion acceleration of the two first data are respectively as follows: 21cm, 5cm, 7cm, 0cm/s, 15cm/s, 2cm/s, 0m/s2、2m/s2、1m/s2If the three-dimensional motion speed of the first data is different from the three-dimensional motion speed of the second first data and the three-dimensional motion acceleration of the first data is different from the three-dimensional motion acceleration of the second first data, it is determined that the first data does not satisfy the similar condition, and the action category of the target user is determined to be a dynamic action.
The step of judging whether each first data satisfies the similar condition by the computer may further include: obtaining a skeletal point position change value of a target user according to each first data, and if the skeletal point position change value does not exceed a first preset threshold, judging that each first data meets a similar condition; and if the position variation value of the skeleton point exceeds a first preset threshold value, judging that the first data do not meet the similar condition.
Illustratively, when the first preset threshold is 2cm, if the position of the skeleton point is changed to 1cm, judging that each piece of first data meets the similar condition, and determining that the action category of the target user is a static posture; and if the position of the skeleton point is changed to 5cm, judging that the first data do not meet the similar conditions, and determining the action type of the target user to be dynamic action.
Similarly, the step of judging whether each first data satisfies the similar condition by the computer may further include: obtaining a bone point speed change value of the target user according to the first data, and if the bone point speed change value does not exceed a second preset threshold, judging that the first data meet a similar condition; and if the speed change value of the bone point exceeds a second preset threshold value, judging that the first data do not meet the similar condition. Or obtaining a bone point acceleration change value of the target user according to the first data, and if the bone point acceleration change value does not exceed a third preset threshold, judging that the first data meet the similar condition; and if the acceleration change value of the bone point exceeds a third preset threshold value, judging that each first data does not meet the similar condition.
Illustratively, when the second preset threshold is 0.5cm/s, if the speed of the bone point changes to 0cm/s, judging that each piece of first data meets the similar condition, and determining that the action category of the target user is a static posture; and if the position of the skeleton point is changed to 5cm/s, judging that the first data do not meet the similar conditions, and determining the action type of the target user to be dynamic action.
The first preset threshold, the second preset threshold and the third preset threshold may be set according to a user requirement, which is not limited in the present invention.
S103, when the action type of the target user is a static posture, determining the target posture of the target user according to each piece of first data, and determining a target musical instrument type corresponding to the target posture according to the target posture and a preset musical instrument type template library; and displaying the pattern corresponding to the target musical instrument type according to the target musical instrument type and the preset pattern library.
The computer may determine the target posture of the target user by performing analysis processing on each piece of the first data by means of signal processing, machine learning, position recognition, position change tracking, or the like. In practical applications, the method for determining the target posture of the target user may be set according to user requirements, which is not limited in the present invention.
In the embodiment of the present invention, the musical instrument type template library may store a correspondence relationship between each musical instrument type and a static posture. For example, the correspondence between each instrument type stored in the instrument type template library and the static posture may be as shown in table 1:
TABLE 1
| Kind of musical instrument | Static posture |
| Drum | Both hands are placed in front of the chest |
| Violin | Both hands lift on the top of the head |
| Trombone | With hands separately held at the top of the head |
The step of determining, by the computer, a target instrument type corresponding to the target posture according to the target posture and the preset instrument type template library may be: searching a static posture which is the same as the target posture in a preset instrument type template library; and determining the instrument type corresponding to the searched static posture according to the instrument type template library, and determining the instrument type as the target instrument type corresponding to the target posture.
Illustratively, each piece of first data is analyzed and processed by using a machine learning method, so that the target posture of the target user is determined to be that two hands lift at the top of the head, the instrument type corresponding to the posture that the two hands lift at the top of the head in a preset instrument type template library is a violin, and the target instrument type is determined to be the violin.
Specifically, the step of displaying, by the computer, the pattern corresponding to the target musical instrument type according to the target musical instrument type and the preset pattern library may be: determining a target pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library; judging whether a displayed pattern exists at present; if no pattern is currently displayed, displaying the target pattern; if the pattern displayed currently exists, judging whether the target pattern is consistent with the pattern displayed currently; if yes, keeping the currently displayed pattern; and if not, displaying the target pattern.
It should be noted that the output of the pattern is performed by the graphic output framework. The graphical interface can output cartoon animal patterns and cartoon scenes, and each cartoon animal can be used for representing a musical instrument type, such as: the lion corresponds to the drum; the fox corresponds to a violin; the rabbit corresponds to the trombone. And displaying the target pattern corresponding to the target musical instrument type through the one-to-one correspondence relationship between the musical instrument type and the cartoon animal pattern. Directly displaying the target pattern under the condition that no pattern is currently displayed; under the condition that a displayed pattern exists currently, whether the target pattern is consistent with the currently displayed pattern needs to be judged, and when the target pattern is consistent with the currently displayed pattern, a signal is sent to a pattern output frame to keep the currently displayed pattern; and when the target pattern is inconsistent with the currently displayed pattern, sending a signal to the graphic output framework, and updating the currently displayed pattern to display the target pattern.
Exemplarily, when the type of the target musical instrument is a violin and the pattern corresponding to the violin in the preset pattern library is a fox, if the pattern is not displayed currently, displaying the fox pattern; if the currently displayed pattern is a fox, keeping displaying the fox pattern; and if the currently displayed pattern is the tiger, displaying the fox pattern.
And S104, when the action type of the target user is a dynamic action, determining the target action of the target user according to each piece of first data, and outputting the audio corresponding to the current display pattern according to the target action, the current display pattern and a preset audio library.
It should be noted that the computer may analyze and process each first data by using pattern recognition, machine learning, finite state automata, and the like, so as to determine the target action of the target user. In practical applications, the manner of determining the target instrument type corresponding to the target action of the target user may be set according to user requirements, which is not limited in the present invention. The specific method for determining the target action of the target user belongs to the prior art, and the detailed description of the method is omitted here.
Illustratively, each first datum is analyzed and processed by pattern recognition, so as to determine the target action of the target user as: one downward motion of the hand.
Specifically, the step of outputting, by the computer, the audio corresponding to the current display pattern according to the target motion, the current display pattern, and the preset audio library may be: determining a target pattern corresponding to the target action according to the corresponding relation between the preset pattern and the action; judging whether the target pattern is consistent with the currently displayed pattern; if so, outputting the audio corresponding to the currently displayed pattern according to a preset audio library.
It should be noted that the target user may perform different musical instruments with different motions. In the embodiment of the invention, the action data of various musical instruments can be collected in advance and marked to form an action template library for identifying the musical instruments, and the action template library can store the corresponding relation between various musical instrument types and actions. For example, the correspondence between each instrument type and the motion stored in the motion template library may be as shown in table 2:
TABLE 2
| Kind of musical instrument | Movement of |
| Drum | Downward movement of the hand |
| Violin | Side-to-side movement of hands |
| Trombone | Fore-and-aft movement of the hand |
Illustratively, the action template library of the embodiment of the present invention stores actions corresponding to three instruments, namely a drum, a violin and a trombone, respectively, and is described as follows:
corresponding to the drum action: in practical applications, the action of knocking the drum is mainly completed by the downward movement of the hand, so the detection is based on the data of the hand of the target user. The target action needs to satisfy two decision criteria:
judging the overall motion, namely judging whether the hand moves downwards or not, and setting a motion duration interval, wherein the motion of the hand from top to bottom in the interval is specified to be in accordance with the standard; the lower limit of the interval is the time that an ordinary person can finish the corresponding tapping action most quickly, and may be 5 milliseconds, 8 milliseconds and the like, and the upper limit is the set time that the ordinary person can finish the tapping action most slowly, and may be 1 second, 1.5 seconds and the like.
When first data is received, the first data is stored and used as reference first data, each first data of which the time difference with the reference first data is within an action time interval is found out in sequence, each first data is traversed according to the sequence of time length from short to long time length, whether the position difference value of the reference first data and the traversed first data is larger than an integral action threshold value or not is judged, and if the position difference value is larger than the threshold value, action triggering is indicated. And the shorter the duration of the interval between the reference first data and the traversed first data is, the faster the action speed of the target user for knocking the drum can be considered, and the information can be fed back to the output audio, and the faster the action speed is, the louder the output audio is.
And judging ending action, namely judging whether the movement distance of the two latest first data is greater than an ending action threshold value after the integral action judgment is met, so that the action trigger is the first trigger of one action instead of stopping in place after being knocked. This determination can effectively prevent false touches where a single tap triggers multiple actions. The ending action threshold may be set according to user requirements, and the present invention is not limited herein.
Corresponding to the action of the violin: in practical applications, the motion of pulling the violin is mainly completed by the left-right movement of the hand, and therefore, the detection is based on the data of the hand of the target user. In the embodiment of the invention, a target user uses a right hand-pulled musical instrument as an example, a finite state automaton is constructed for the state of the right hand, the automaton performs state transition after the corresponding state is met, and simultaneously, a control signal (note start/note end) is sent out.
The five states are defined as follows: s1-right hand in right side preparation area preparation start, S2-right hand in right to left movement, S3-right hand in left side preparation area preparation start, S4-right hand in left to right movement, S5-right hand in other areas. The key to the state transition is: 1) the state S2 is entered only in the state S1, the state S2 is entered, the state S4 is entered only in the state S3, and the state S4 is entered, a note start signal is issued. In other words, the event of pulling the piano to the left can be triggered only in the preparation area on the right side, and the processing method can filter out false triggering generated after the action is finished; 2) in the state of S2, the motion must be continued to keep in the state of S2, and in the same way, in the state of S4, the motion must be continued to keep in the state of S4, otherwise, the event is considered to be over, and a note over signal is sent out. The states of S1, S3, and S5 are three states in parallel, respectively corresponding to three regions in a plane, and automatically switch among the three states of S1, S3, and S5 based on the right-hand position data in the acquired first data without triggering an operation.
Actions corresponding to the trombone: in practical application, the motion of pulling the trombone is mainly completed by the back and forth motion of the hand, which is similar to the left and right motion of a violin, so a finite state automaton is also adopted, only the data of the X-Y plane of the depth data in the area corresponding to the states S1, S3 and S5 are converted into the data of the X-Z plane, and the specific method refers to the description of the violin part, and is not repeated herein.
The computer can determine a target instrument type corresponding to the target motion according to the motion template library, and can determine a target pattern corresponding to the target motion according to the target instrument type. Thus, it is determined whether the target pattern is identical to the currently displayed pattern, and a signal is sent to the audio output frame as a result of the determination, so that the audio output frame determines whether to output audio. When the target pattern is consistent with the currently displayed pattern, sending a signal to an audio output frame, and outputting an audio corresponding to the currently displayed pattern; and when the target pattern is inconsistent with the currently displayed pattern, sending a signal to the audio output frame, and not outputting the audio.
The audio output framework may use a MIDI (Musical Instrument Digital Interface) framework of the Windows system. The specific method for outputting audio by using the MIDI frame belongs to the prior art, and the embodiments of the present invention are not described herein again.
Illustratively, according to the motion template library, it is determined that the instrument type corresponding to the target motion of the target user is a violin, the currently displayed pattern is a fox, and the motion corresponding to the fox pattern is a motion for playing the violin, so that it is determined that the target pattern corresponding to the target motion is a fox, which is identical to the currently displayed pattern, and thus, violin audio corresponding to the fox pattern is output.
Therefore, the interactive musical instrument playing method provided by the embodiment of the invention realizes the interactive musical instrument playing.
Corresponding to the method embodiment, the embodiment of the invention also provides an interactive musical instrument playing device.
Referring to fig. 2, fig. 2 is a schematic structural diagram of an interactive musical instrument playing apparatus according to an embodiment of the present invention, including:
an obtaining module 201, configured to obtain a preset number of first data of a target user within a preset time duration;
a determining module 202, configured to determine an action category of the target user according to each first data, where the action category includes a static gesture or a dynamic action;
the display module 203 is configured to determine, when the determination module determines that the action category of the target user is a static posture, a target posture of the target user according to each first data, and determine, according to the target posture and a preset instrument category template library, a target instrument category corresponding to the target posture; displaying a pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library;
an output module 204, configured to, when the determining module determines that the action category of the target user is a dynamic action, determine a target action of the target user according to each first data, and output an audio corresponding to the current display pattern according to the target action, the current display pattern, and a preset audio library.
Wherein, the determining module 202 includes:
a first judging submodule (not shown in the figure) for judging whether each of the first data satisfies a similar condition;
a first determining sub-module (not shown in the figure) for determining the action category of the target user as a static gesture when the first determining sub-module determines that the action category is a static gesture;
and a second determining sub-module (not shown in the figure) configured to determine that the action category of the target user is a dynamic action when the first determining sub-module determines that the result is negative.
Wherein, the musical instrument category template library stores the corresponding relationship between each musical instrument category and the static posture, and the display module 203 may include:
a searching sub-module (not shown in the figure) for searching the static posture identical to the target posture in the preset musical instrument category template library;
and a third determining sub-module (not shown in the figure) configured to determine, according to the musical instrument category template library, a musical instrument category corresponding to the found static posture, and determine the musical instrument category as a target musical instrument category corresponding to the target posture.
The display module 203 may further include:
a fourth determining sub-module (not shown in the figure) for determining a target pattern corresponding to the target musical instrument type according to the target musical instrument type and a preset pattern library;
a second judgment sub-module (not shown in the figure) for judging whether there is a pattern displayed currently;
a first display sub-module (not shown in the figure) for displaying the target pattern when the second judgment sub-module judges that the result is negative;
a second display sub-module (not shown in the figure), configured to determine whether the target pattern is consistent with the currently displayed pattern when the determination result of the second determination sub-module is yes; if yes, maintaining the currently displayed pattern; and if not, displaying the target pattern.
The output module 204 includes:
a fifth determining sub-module (not shown in the figure) for determining a target pattern corresponding to the target action according to a corresponding relationship between preset patterns and actions;
a third judging sub-module (not shown in the figure) for judging whether the target pattern is consistent with the currently displayed pattern;
and an output sub-module (not shown in the figure) configured to output an audio corresponding to the currently displayed pattern according to the preset audio library when the determination result of the third determining sub-module is yes.
Therefore, the interactive musical instrument playing device provided by the embodiment of the invention realizes the interactive musical instrument playing.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (4)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611108803.4A CN106601217B (en) | 2016-12-06 | 2016-12-06 | A kind of interactive musical instrument performance method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611108803.4A CN106601217B (en) | 2016-12-06 | 2016-12-06 | A kind of interactive musical instrument performance method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106601217A CN106601217A (en) | 2017-04-26 |
| CN106601217B true CN106601217B (en) | 2021-03-02 |
Family
ID=58596507
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611108803.4A Active CN106601217B (en) | 2016-12-06 | 2016-12-06 | A kind of interactive musical instrument performance method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106601217B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111199455A (en) * | 2019-12-19 | 2020-05-26 | 南京酷派软件技术有限公司 | Method, apparatus, electronic device and medium for selecting musical instrument |
| CN111103983A (en) * | 2019-12-31 | 2020-05-05 | 维沃移动通信有限公司 | AR musical instrument playing method and electronic equipment |
| CN112752149B (en) * | 2020-12-29 | 2023-06-06 | 广州繁星互娱信息科技有限公司 | Live broadcast method, live broadcast device, terminal and storage medium |
| CN114115528B (en) * | 2021-11-02 | 2024-01-19 | 深圳市雷鸟网络传媒有限公司 | Virtual object control method, device, computer equipment and storage medium |
| CN116386424A (en) * | 2023-04-12 | 2023-07-04 | 黄志坚 | Method, device and computer readable storage medium for music teaching |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101504832A (en) * | 2009-03-24 | 2009-08-12 | 北京理工大学 | Virtual performance system based on hand motion sensing |
| CN102478959A (en) * | 2010-11-28 | 2012-05-30 | 蒋霞 | Control system and method for electronic device |
| CN103986825A (en) * | 2014-05-09 | 2014-08-13 | 苏州乐聚一堂电子科技有限公司 | Image special effect system of mobile personalized karaoke |
| CN104007844A (en) * | 2014-06-18 | 2014-08-27 | 原硕朋 | Electronic instrument and wearable type input device for same |
| CN205680441U (en) * | 2016-06-06 | 2016-11-09 | 北京千音互联科技有限公司 | A kind of virtual performance system |
| CN106157363A (en) * | 2016-06-28 | 2016-11-23 | 广东欧珀移动通信有限公司 | A camera method, device and mobile terminal based on augmented reality |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6720949B1 (en) * | 1997-08-22 | 2004-04-13 | Timothy R. Pryor | Man machine interfaces and applications |
| US20020036617A1 (en) * | 1998-08-21 | 2002-03-28 | Timothy R. Pryor | Novel man machine interfaces and applications |
| US9898675B2 (en) * | 2009-05-01 | 2018-02-20 | Microsoft Technology Licensing, Llc | User movement tracking feedback to improve tracking |
| CN102117117A (en) * | 2010-01-06 | 2011-07-06 | 致伸科技股份有限公司 | System and method for controlling user gestures by using image capture device |
| JP6074170B2 (en) * | 2011-06-23 | 2017-02-01 | インテル・コーポレーション | Short range motion tracking system and method |
| CN102609093A (en) * | 2012-02-16 | 2012-07-25 | 中国农业大学 | Method and device for controlling video playing by using gestures |
| CN103235641B (en) * | 2013-03-17 | 2016-08-03 | 浙江大学 | Dummy keyboard musical instrument system that a kind of 6 dimension sense organs are mutual and its implementation |
| CN105915987B (en) * | 2016-04-15 | 2018-07-06 | 济南大学 | A kind of implicit interactions method towards smart television |
-
2016
- 2016-12-06 CN CN201611108803.4A patent/CN106601217B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101504832A (en) * | 2009-03-24 | 2009-08-12 | 北京理工大学 | Virtual performance system based on hand motion sensing |
| CN102478959A (en) * | 2010-11-28 | 2012-05-30 | 蒋霞 | Control system and method for electronic device |
| CN103986825A (en) * | 2014-05-09 | 2014-08-13 | 苏州乐聚一堂电子科技有限公司 | Image special effect system of mobile personalized karaoke |
| CN104007844A (en) * | 2014-06-18 | 2014-08-27 | 原硕朋 | Electronic instrument and wearable type input device for same |
| CN205680441U (en) * | 2016-06-06 | 2016-11-09 | 北京千音互联科技有限公司 | A kind of virtual performance system |
| CN106157363A (en) * | 2016-06-28 | 2016-11-23 | 广东欧珀移动通信有限公司 | A camera method, device and mobile terminal based on augmented reality |
Non-Patent Citations (3)
| Title |
|---|
| "A Motion Recognition Method for a Wearable Dancing Musical Instrument";Minoru Fujimoto et al.;《2009 International Symposium on Wearable Computers》;20091231;全文 * |
| "Hand Gesture Recognition To Understand Musical Conducting Action";Hongmo Je et al.;《16th IEEE International Conference on Robot & Human Interactive Communication》;20070829;全文 * |
| "交互式乐器演奏的六自由度力觉渲染方法";童号等;《中国图象图形学报》;20150228;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106601217A (en) | 2017-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106601217B (en) | A kind of interactive musical instrument performance method and device | |
| US9030498B2 (en) | Combining explicit select gestures and timeclick in a non-tactile three dimensional user interface | |
| TWI411935B (en) | System and method for generating control instruction by identifying user posture captured by image pickup device | |
| CN104090652A (en) | Voice input method and device | |
| WO2020059245A1 (en) | Information processing device, information processing method and information processing program | |
| CN106393113A (en) | Robot and interactive control method for robot | |
| EP4307096A1 (en) | Key function execution method, apparatus and device, and storage medium | |
| CN106774028A (en) | A kind of robot control method and device based on time shaft | |
| JP6353660B2 (en) | Sign language word classification information generation device and program thereof | |
| KR101567154B1 (en) | Method for processing dialogue based on multiple user and apparatus for performing the same | |
| CN117980873A (en) | Display device and control method thereof | |
| KR101652705B1 (en) | Apparatus for predicting intention of user using multi modal information and method thereof | |
| CN110007748B (en) | Terminal control method, processing device, storage medium and terminal | |
| Bordegoni et al. | A dynamic gesture language and graphical feedback for interaction in a 3d user interface | |
| Rustagi et al. | Virtual control using hand-tracking | |
| JP2020017283A (en) | Program, information processing apparatus and information processing method | |
| JP6279635B2 (en) | Program, information processing apparatus and information processing method | |
| CN114610153A (en) | A display device and dynamic gesture interaction method | |
| JP5676959B2 (en) | Information processing apparatus and control method thereof | |
| CN206475183U (en) | Robot | |
| CN114510142B (en) | Gesture recognition method based on two-dimensional image, gesture recognition system based on two-dimensional image and electronic equipment | |
| US12216830B2 (en) | Information processing apparatus and information processing method | |
| JP2019144822A (en) | Explicit knowledge formalization system and method of the same | |
| KR101370830B1 (en) | System and Method for Implementing User Interface | |
| US20100185570A1 (en) | Three-dimensional motion identifying method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |