JP2020046500A - Information processing apparatus, information processing method and information processing program - Google Patents

Abstract

To provide an information processing apparatus, an information processing method and an information processing program capable of acquiring information representing a musical performance of a musical instrument from an image.SOLUTION: An information processing apparatus 100 comprises: a position recognition unit which recognizes the position of a region of the body of a player from an input image; a musical instrument recognition unit which recognizes the musical instrument from the input image; and a musical performance information generation unit which generates, based upon relativity between the position of the region and the musical instrument, musical performance information representing a musical performance of the musical instrument by the player.SELECTED DRAWING: Figure 2

Description

  The present technology relates to an information processing device, an information processing method, and an information processing program.

  2. Description of the Related Art Hitherto, a system has been proposed in which performance of a person such as a dance is converted into data (Patent Document 1).

JP 2016-24740 A

  The system described in Patent Literature 1 generates musical score data in which a dance motion of a performer in a three-dimensional space is recorded. In the method of converting such an operation into data, it is difficult to apply the operation to other operations, for example, a musical instrument performance as it is, because information and processing required for data conversion differ depending on the type of operation.

  The present technology has been made in view of such a point, and an object of the present technology is to provide an information processing apparatus, an information processing method, and an information processing program capable of acquiring information indicating performance of a musical instrument from an image.

  In order to solve the above-described problem, a first technique includes a position recognition unit that recognizes a position of a body part of a player from an input image, a musical instrument recognition unit that recognizes a musical instrument from the input image, and a position of the part. An information processing apparatus comprising: a performance information generating unit configured to generate performance information indicating a performance of a musical instrument by a player based on relevance to the musical instrument.

  The second technique recognizes a position of a body part of a player from an input image, recognizes a musical instrument from the input image, and plays a musical instrument by the player based on the relationship between the position of the part and the musical instrument. This is an information processing method for generating performance information shown in FIG.

  Further, a third technique recognizes the position of a body part of a player from an input image, recognizes a musical instrument from the input image, and performs the performance of the musical instrument by the player based on the relationship between the position of the part and the musical instrument. Is an information processing program for causing a computer to execute an information processing method for generating performance information indicating the following.

FIG. 3 is a block diagram illustrating a configuration of a terminal device. FIG. 2 is a block diagram illustrating a configuration of the information processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of an input image according to the first embodiment. It is explanatory drawing about recognition of a player's hand. It is explanatory drawing about recognition of a player's hand. It is explanatory drawing about recognition of a chord (chord). 6 is a flowchart illustrating a process of generating partial performance information according to the first embodiment. 6 is a flowchart illustrating a process of generating composite performance information according to the first embodiment. FIG. 14 is a diagram illustrating an example of an input image according to the second embodiment. It is a block diagram showing the composition of the information processor concerning a 2nd embodiment. It is a flow chart which shows generation processing of partial performance information in a 2nd embodiment. 15 is a flowchart illustrating a process of generating composite performance information according to the second embodiment. FIG. 14 is a diagram illustrating an example of an input image according to the third embodiment. It is a block diagram showing the composition of the information processor concerning a 3rd embodiment. It is a flowchart which shows the production | generation process of the partial performance information in 3rd Embodiment. It is a flow chart which shows generation processing of compound performance information in a 3rd embodiment.

Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order.
<1. First Embodiment>
[1-1. Configuration of Terminal Device]
[1-2. Configuration of Information Processing Apparatus]
[1-3. Processing by information processing device]
[1-3-1. Generation of partial performance information]
[1-3-2. Generating composite performance information]
<2. Second Embodiment>
[2-1. Configuration of Information Processing Apparatus]
[2-2. Processing of information processing device]
<3. Third Embodiment>
[3-1. Configuration of Information Processing Apparatus]
[3-2. Processing of information processing device]
<4. Modification>

<1. First Embodiment>
[1-1. Configuration of Terminal Device]
First, the terminal device 10 will be described with reference to FIG. The terminal device 10 includes a control unit 11, a storage unit 12, a communication unit 13, a display unit 14, an input unit 15, a camera unit 16, and an information processing device 100.

  The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The ROM stores programs and the like that are read and operated by the CPU. The RAM is used as a work memory of the CPU. The CPU controls the entire terminal device 10 by executing various processes in accordance with the programs stored in the ROM and issuing commands.

  The storage unit 12 is a large-capacity storage medium using, for example, a hard disk, a semiconductor memory, or the like. The storage unit 12 can store a captured image and a captured video captured by the camera unit 16, performance information and music score information generated by the information processing apparatus 100, as well as contents and applications.

  The communication unit 13 is a communication module for communicating with another device, the Internet, or the like, a communication connector, or the like. The communication by the communication unit 13 may be wired communication such as USB communication, wireless LAN such as Wi-Fi, wireless communication such as Bluetooth (registered trademark), ZigBee, 4G (4th generation mobile communication system), or broadband.

  The display unit 14 is a display device including, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), and an organic EL (Electro Luminescence) panel. The display unit 14 displays a user interface of the terminal device 10, an interface presented to the user for processing by the information processing device 100, and the like.

  The input unit 15 receives a user's operation input to the terminal device 10. When an input is made from the user to the input unit 15, an input signal corresponding to the input is generated and output to the control unit 11. Then, the control unit 11 performs arithmetic processing corresponding to the input signal and controls the terminal device 10. As the input unit 15, a touch panel integrated with the display unit 14, a pointing device that operates by tracing a flat sensor that is not integrated with the display unit 14 called a trackpad or a touchpad with a finger, There are keyboard, mouse, etc.

  The camera unit 16 includes an image sensor, an image processing LSI, and the like, and has a camera function capable of capturing still images and videos. The still image or video captured by the camera unit 16 can be used for performance information generation processing in the information processing device 100. The camera unit 16 is not an essential component of the terminal device 10.

  The information processing device 100 performs performance information generation processing according to the present technology. Details of the information processing device 100 will be described later.

  The terminal device 10 is configured as described above. Specific examples of the terminal device 10 include a personal computer, a notebook computer, a tablet terminal, a smartphone, an electronic keyboard, a synthesizer, and a DAW (Digital Audio Workstation).

[1-2. Configuration of Information Processing Apparatus]
Next, the configuration of the information processing apparatus 100 will be described with reference to FIG. The information processing apparatus 100 includes an image input unit 101, a position recognition unit 102, a shape recognition unit 103, a motion recognition unit 104, a musical instrument recognition unit 105, a relevance recognition unit 106, a performance information generation unit 107, and a score information generation unit 108. It is configured.

  The image input unit 101 receives a plurality of continuous still images as input images to be processed or a plurality of continuous frame images forming a moving image. The image input unit 101 supplies the input image to the position recognition unit 102 and the musical instrument recognition unit 105. The input image to be processed in the present technology is each of a plurality of continuous still images and each of a plurality of frame images forming a moving image.

  The input image may be an image captured by the camera unit 16 provided in the terminal device 10, or may be an image captured by a camera other than the camera unit 16 and captured by the information processing device 100 via the terminal device 10. Further, the information may be supplied from another external device to the information processing device 100 via the terminal device 10. Further, it may be a photograph of a performance actually performed in front of the eyes of the user of the information processing apparatus 100, or a photograph of a video displayed on a display such as a television or a personal computer. Further, a commercially available DVD, a video recorded on Blue ray (registered trademark), a still image or a video that can be obtained on the Internet, or the like may be used. That is, the input image may be any image as long as the image shows a state where the player is performing. The input image may be an IR image or the like in addition to the RBG (Red, Green, Blue) image.

  As shown in FIG. 3, the input image according to the first embodiment shows the entire area (performance area) for the performance in which both hands of the player and the hand of the player touch on the instrument played by the player. Is what it is.

  The position recognition unit 102 performs a human body hand recognition technique such as Hand Pose Detection, Hand Pose Estimation, and Hand segmentation from an input image, a feature point extraction method such as HOG (Histogram of Oriented Gradient), SIFT (Scale Invariant Feature Transform), A three-dimensional hand, which is a part of the player's body in an input image, is obtained by a subject recognition method using pattern recognition such as Boosting or SVM (Support Vector Machine), a region extraction method using Graph Cut, or the like, or a CNN (Convolutional Neural Network). Recognize the position. The position recognizing unit 102 also recognizes finger positions, arm positions, elbow positions, and the like as parts of the player's body as necessary for generating performance information, in addition to the hands. The three-dimensional hand position information is supplied to the shape recognition unit 103, the motion recognition unit 104, and the relevance recognition unit 106.

The characteristic points of the hand for recognizing the three-dimensional position of the hand include a fingertip, a finger joint, and a wrist. Since the position information is information indicating the three-dimensional position of the player's hand in the input image, for example, the coordinates of (x, y, z) where the predetermined position of the input image is the origin (0, 0, 0) It is represented by If the number of the continuous input image is t (t = 1, 2, 3,...) And the hand feature point is P (P = 1, 2, 3,...), The position information is (x _tP , _ytP , _ztP ).

For example, as shown in FIG. 4A, when five feature points of the hand are recognized in the input image (t = 1),
Feature point _{P1: (x 11, y 11} , z 11)
Feature point _{P2: (x 12, y 12} , z 12)
Feature point _{P3: (x 13, y 13} , z 13)
Feature point _{P4: (x 14, y 14} , z 14)
Feature point _{P5: (x 15, y 15} , z 15)
It is represented as

Also, as shown in FIG. 4B, in the input image (t = 2), when five hand feature points are recognized,
Feature point _{P1: (x 21, y 21} , z 21)
Feature point _{P2: (x 22, y 22} , z 22)
Feature point _{P3: (x 23, y 23} , z 23)
Feature point _{P4: (x 24, y 24} , z 24)
Feature point _{P5: (x 25, y 25} , z 25)
It is represented as

  Note that the three-dimensional position information of the hand may be a global coordinate system of the camera origin, or a local coordinate system + depth information on the input image. Alternatively, the three-dimensional position of the hand may be obtained using the center of gravity and Depth information of the region obtained by Hand Segmentation.

  Note that FIG. 4 is a diagram in which characteristic points are recognized at the tips of five fingers of the hand for convenience of description, but in reality, as shown by a plurality of black dots superimposed on the hand in FIG. For example, a number of feature points such as a joint part, a web part, and a wrist of each finger are recognized. Recognition of a large number of feature points enables more accurate performance information to be generated.

Further, in a cut-out image obtained by cutting out a partial area of the input image, a predetermined position of the cut-out image, which is a coordinate system different from the (x, y, z) coordinate system, is set as an origin ( _utP , _vtP). , d _tP ).

  The shape recognizing unit 103 recognizes the hand shape indicated by the position information supplied from the position recognizing unit 102 using a technique such as CNN, pattern matching, and boosting. The hand shape information is supplied to the motion recognition unit 104 and the performance information generation unit 107.

  The motion recognition unit 104 recognizes the motion of the player's hand whose position and shape have been recognized using techniques such as CNN and Hand Tracking. The hand movement information is supplied to the performance information generation unit 107. The hand movement can be recognized from a change in motion vector between one input image (t) of a plurality of continuous input images and an input image (t + n) subsequent to the input image (t) in time series. it can.

  The musical instrument recognizing unit 105 recognizes a musical instrument in an input image and a performance area (performance area) in which the player's hand touches the musical instrument in the input image using techniques such as CNN and pattern matching. The performance area is, for example, a keyboard when the musical instrument is a piano, a pickup portion (a sound hole when the musical instrument is an acoustic guitar) and a neck when the musical instrument is a guitar. The instrument recognition information is supplied to the association recognition unit 106.

  The relevance recognition unit 106 recognizes the relevance between the position of the player's hand and the playing area of the musical instrument using techniques such as CNN and pattern matching. The relevance is a relation between a player and a musical instrument for playing a musical instrument, that is, a contact position indicating where the player's hand is in contact with the musical instrument playing area. The relevance is the direction of movement of the player's hands, arms, elbows, and other parts with respect to the performance area of the instrument. The relevancy information is supplied to the performance information generation unit 107.

  The performance information generation unit 107 recognizes whether or not the player is in a playing state using a technique such as CNN. Then, in the input image, a performance element based on a state in which the player is performing (first performance element), a performance element based on a state in which the player is not performing (second performance element), and a performance spanning a plurality of input images Performance information (partial performance information) corresponding to one input image is generated from the element (third performance element).

  Although the first performance element varies depending on the musical instrument, a keyboard instrument such as a piano has a scale, a sound length, a tempo, a strength, and the like. Similarly, a stringed instrument such as a guitar has a scale, a sound length, and a sound intensity. Further, for percussion instruments such as drums, there are a type of drum set to be hit, a sound length, a tempo, a strength and the like.

  Further, as the second performance element, for any musical instrument, there is a length of rest. As the third performance element, there are a tempo, a sound length, a rest length, a key, a sound intensity, and the like. The dynamics of the sound, the duration of the sound, etc. are also the first performance element and the third performance element. In some cases, the strength and duration of the sound can be estimated from one input image. If there is, a plurality of input images may be required for estimation. For example, when the player's finger is located far away from the musical performance area of the musical instrument in one input image, it can be estimated from the single input image that the sound is strong. In the case where the image moves finely near the performance area, the strength cannot be estimated with one image, and it is necessary to estimate the strength by recognizing the movement of the player's finger with reference to a plurality of input images. .

  As a method for acquiring a performance element from the relevance information between the position of the player's hand and the playing area of the instrument, the hand position recognized by the position recognition unit 102 and the hand shape recognized by the shape recognition unit 103 Then, on the basis of the musical instrument recognized by the musical instrument recognizing unit 105 and the performance area, it recognizes which keyboard of the musical instrument's performance area the finger of the player touches. Thus, it is possible to recognize which sound of the musical scale is sounding due to the performance in the state of the player in the input image. It is also possible to recognize what kind of chord (chord) composed of a plurality of sounds is sounding.

  In addition, by recognizing how long the finger of the player keeps touching the same place in the performance area, the sound length can be recognized.

  For example, if the musical instrument is a guitar, the chord (chord) is recognized by the performance information generation unit 107 as shown in FIGS. 6A and 6B. A plurality of images are stored for each type of chord (chord). Then, by comparing (template matching) the finger position information and the finger shape information extracted from the input image with the template image (template matching), a chord (chord) having the closest finger position and shape is determined.

  As shown in FIG. 6C, the recognition of chords (chords) is performed in advance by the performance information generation unit 107 by transmitting coordinate information of finger characteristic points indicating finger positions when playing chords (chords) to the chords (chords). It is also possible to hold a plurality of types for each type and compare the coordinate information with the position information (coordinate information) of the finger extracted from the input image.

  Also, based on the hand movement recognized by the motion recognition unit 104, a performance that can be recognized from one input image (t) and an input image (t + n) subsequent to the input image (t) in time series. It is possible to recognize whether or not the player is performing from the movement of the user's hand in a substantially vertical direction, the strength of the performance, the tempo, and the like.

  The substantially vertical direction in this case is a direction substantially perpendicular to the direction in which the keyboards are arranged when the musical instrument is a piano. Whether or not the player is playing can be determined based on whether or not the hand is off the keyboard of the musical instrument. The strength of the performance can be determined from the position of the hand (hand height) in a substantially vertical direction. For example, it can be determined that the sound is stronger when the hand is farther away from the keyboard in the vertical direction, and the sound is weaker when the hand is closer to the keyboard in the vertical direction. Further, the tempo of the music can be recognized from the time interval of the regular up and down movement in the vertical direction of the hand. In this way, in order to recognize performance elements related to time, such as the tempo of a song and the duration of a sound, the frame rate constituting the video is associated with real time, and the player's regular movements in real time are It can be obtained from the operation interval and the video playback time.

  Similarly, the scale is recognized from the movement of the player's hand in a substantially horizontal direction that can be recognized from one input image (t) and the input image (t + n) subsequent to the input image (t) in time series. can do. In this case, the substantially horizontal direction is a direction substantially horizontal to the direction in which the keyboards are arranged when the musical instrument is a piano. Specifically, by changing the position of the hand with respect to the piano in a substantially horizontal direction, it is possible to know which area of the piano's keyboard is being played, thereby recognizing the scale being played, such as changes in the range and octave. it can.

  The movement of the hand in the substantially vertical direction and the movement in the substantially horizontal direction include the input image (t) of one of a plurality of continuous input images and the input image (t + n) subsequent to the input image (t) in time series. Can be recognized from the change of the motion vector.

  The third performance element can be generated based on a change in a player's finger or arm over a plurality of input images. For example, it can be determined that the longer the time the hand is separated from the keyboard in the vertical direction over a plurality of input images, the stronger the sound to be played next is, and thus the third performance element can be used. In addition, the tempo of the music can be recognized from the time interval of the vertical movement in the vertical direction of the hand straddling a plurality of input images, and can be used as the third performance element.

  Further, the performance information generation unit 107 searches partial performance information corresponding to each of the plurality of input images in chronological order, thereby forming a composite performance of a phrase composed of the plurality of input images, a part or all of a song. Generate information. The composite performance information of a part of a phrase or a piece of music is performance information in units of one or more measures.

  The partial performance information and the composite performance information are not limited to music notation written in staff notation. What kind of information can be used if a player, a computer, software for music performance, software for music creation, etc. can reproduce music based on the information. Format information may be used. For example, information in a MIDI (Musical Instrument Digital Interface) format, information in a programming format, information in a format unique to music playing / production software, and the like may be used.

  When the partial performance information is supplied from the performance information generating section 107, the musical score information generating section 108 generates partial musical score information corresponding to one input image. When composite performance information is supplied from the performance information generation unit 107, composite musical score information that is the musical score information of a part or all of a phrase composed of a plurality of input images and music is generated. The musical score referred to here is a musical score written in a staff notation, and information constituting the musical score information includes a note, a rest, a time signature, a tempo, an accidental key, a key signature, strength and the like. The accidental information can be derived from the musical scale being played, which is the first performance element based on the state in which the player is playing, and the key, which is the third performance element spanning a plurality of input images.

  The information processing device 100 is configured as described above. The information processing apparatus 100 is configured by a program, and the program may be installed in the terminal device 10 in advance, or may be downloaded, distributed on a storage medium, or the like, and may be installed in the terminal device 10 by the user. . The information processing apparatus 100 may be realized not only by a program but also by a combination of a dedicated device, a circuit, and the like using hardware having the function.

[1-3. Processing by information processing device]
[1-3-1. Generation of partial performance information]
Next, the flow of processing in the information processing apparatus 100 will be described with reference to the flowchart in FIG. The processing of the flowchart in FIG. 7 is for generating partial performance information corresponding to one input image. Note that, as described above, one input image is one of a plurality of continuous still images or one of a plurality of frame images constituting a moving image.

  First, in step S101, an input image is input to the image input unit 101. This input image may be a single still image or a frame image, a plurality of continuous still images, or a plurality of continuous frame images constituting a moving image. When a plurality of input images are input, the following processes from step S102 are first performed on the first input image (t = 1). When a plurality of continuous still images or a plurality of continuous frame images constituting a moving image are input, the user may be able to select which input image to generate partial performance information for.

  Next, in step S102, the three-dimensional position of the player's hand in the input image is recognized by the position recognition unit 102, and the hand position information is supplied to the shape recognition unit 103, the motion recognition unit 104, and the association recognition unit 106. .

  Next, in step S103, the shape of the hand whose position has been recognized in the input image is recognized by the shape recognition unit 103. The hand shape information is supplied to the motion recognition unit 104 and the performance information generation unit 107. Further, in step S104, the movement recognition unit 104 recognizes the movement of the hand whose position and shape have been recognized. The hand movement information is supplied to the performance information generation unit 107.

  Next, in step S105, the musical instrument and the playing area in the input image are recognized by the musical instrument recognition unit 105. The musical instrument information and the performance area information are supplied to the association recognizing unit 106. The recognition processing of the position, shape, and movement of the player's hand in steps S102 to S104 and the recognition of the musical instrument and the playing area in step S105 may be performed in parallel. You may go first.

  Next, in step S106, the relevance recognizing unit 106 recognizes the relevance of each finger of the hand and the position of the playing area of the corresponding musical instrument. The relevance indicates where the player's hand is located in the performance area of the musical instrument, and the relevancy information is supplied to the performance information generation unit 107.

  Next, in step S107, the performance information generation unit 107 determines from the hand movement information and the relevance information whether or not the player is playing a musical instrument in the input image.

  If the result of the determination is that the player is performing, the process proceeds from step S108 to step S109 (Yes in step S108). Then, in step S109, the performance information generation unit 107 generates a first performance element from the three-dimensional position information of the hand, hand shape information, hand movement information, and relevance information.

  On the other hand, if the result of determination in step S107 is that the player is not playing, the process proceeds from step S108 to step S110 (No in step S108). Then, in step S110, the performance information generation unit 107 generates a second performance element.

  Next, in step S111, the performance information generation unit 107 generates partial performance information corresponding to the input image from the first performance element or the second performance element. Then, in step S112, the partial performance information is output.

  The output partial performance information can be displayed on the display unit 14 of the terminal device 10 or used in music performance software, music production software, or the like provided in the terminal device 10. Further, the musical score information generating unit 108 may generate musical score information based on the performance information in response to a request from a user, a player, or the like. Further, the partial performance information can be stored in the storage unit 12 of the terminal device 10 and read out from the storage unit 12 and used as needed.

  The performance information generation processing for the input image is performed as described above.

[1-3-2. Generating composite performance information]
Next, the flowchart of FIG. 8 will be described. The process of the flowchart of FIG. 8 is a process of generating composite performance information that is performance information of a phrase or a part or all of a song composed of a plurality of input images.

  First, in step S101, a plurality of continuous still images or a plurality of continuous frame images forming a moving image are input to the image input unit 101 as input images. When a plurality of input images are input, the processing from step S102 onward is first performed on the first input image of the input image (t = 1). When a plurality of continuous still images or a plurality of continuous frame images constituting a moving image are input, the user may be able to select from which input image to start processing.

  The processing from step S101 to step S111 is the same as that in the flowchart in FIG.

  After step S111, next in step S121, the performance information generation unit 107 determines whether there is a third performance element that is a performance element extending over a plurality of input images. Whether there is a third performance element straddling a plurality of input images can be determined as follows. For example, in the case of sound intensity (loudness), the strength of the performance recognized in the input image (t) currently being processed is higher than the strength of the performance recognized in the input image (t-1) which is the immediately preceding input image. When it becomes stronger, a performance element "gradually stronger" can be derived from the input image (t-1) to the input image (t). Similarly, for example, when the strength of the performance recognized at the pitch of the input image (t + 1) is greater than the strength of the input image (t), the input image (t−1), the input image (t), and the input image With (t + 1), a performance element of "gradually stronger" can be derived. As described above, when the performance element in the past input image is recognized based on the performance element in the current input image based on the state of each of the plurality of input images to be processed, the “performance element spanning between frames, There is a performance element. "

  If there is a third performance element extending over a plurality of input images, the process proceeds to step S122 (Yes in step S121), and the partial performance information is added by adding the third performance element to the partial performance information generated in step S111. Update. Then, the process proceeds from step S122 to step S123. The third performance element may be a component of the partial performance information in the partial performance information similarly to the first performance element and the second performance element, or may be associated with the partial performance information by associating it with the separate performance information. Is also good.

  On the other hand, if there is no third performance element spanning a plurality of images in step S121, the process proceeds to step S123 (No in step S121).

  Next, in step S123, it is determined whether there is a next input image to be processed. If there are still unprocessed images in a plurality of continuous still images or a plurality of continuous frame images constituting a moving image input to the image input unit 101 in step S101, it is determined that there is a next input image. Returns to step S102 (Yes in step S123). Then, the processes of steps S102 to S123 are performed on the input image in the next order in a time series (if it is a frame image, the image of the next frame number). Steps S102 to S123 are repeated until the processing is performed on each of all the input images that have been input.

  If there is no image to be processed in step S123, the process proceeds to step S124 (No in step S123).

  Next, in step S124, the performance information generation unit 107 connects the partial performance information corresponding to each of the plurality of input images in a time series, thereby forming a phrase, a part of a song, or a part of the plurality of input images. Generate all composite performance information.

  Next, in step S125, the performance information generation unit 107 outputs composite performance information. The output composite performance information can be displayed on the display unit 14 of the terminal device 10 or used in music performance software, music production software, or the like provided in the terminal device 10. Further, the score information generating unit 108 may generate the composite score information based on the composite performance information in response to a request from a user, a player, or the like. Further, when the composite performance information is output, the partial performance information may also be output.

The processing according to the first embodiment is performed as described above. According to the first embodiment of the present technology, it is possible to generate performance information and score information based on a plurality of continuous still images or a plurality of frame images forming a moving image.
As a result, even people who do not have specialized knowledge can easily obtain performance information and score information. Also, for example, performance information and musical score information can be generated based on video data having no sound, video data having deteriorated / damaged sound, and the like. Further, even in an environment where audio cannot be output, performance information can be generated based only on video data.

  In the first embodiment, the input image for generating the performance information is, for example, when the musical instrument is a piano, captured from above, which can recognize both the keyboard, which is the performance area of the piano, and both hands of the player. Are preferred. When the musical instrument is a guitar, it is preferable that the image is taken from the front which can recognize both hands of the pickup part (sound hole in the case of acoustic guitar) which is the playing area of the guitar and the neck and the playing area.

  The present technology can be used for scores of improvisations performed by one or other players, score creation for musical instrument practice, score creation for playing a favorite artist song, composition, arrangement, and the like. Also, when composing or arranging, it is possible to try various performances, phrases, and the like with musical instruments, and to obtain necessary performance patterns or all performance patterns as performance information and score information. Also, there is no need to repeat the steps of “writing a score and playing with an instrument” or “playing with an instrument and writing a score if it is good”.

<2. Second Embodiment>
[2-1. Configuration of Information Processing Apparatus]
Next, a second embodiment of the present technology will be described. In the second embodiment, as shown in FIG. 9, performance information is generated when a part of a hand, which is a part of a player's body, is covered or hidden in an input image. is there. In FIG. 9, a part of the player's left hand is hidden. Note that the configuration of the terminal device 10 on which the information processing device 100 operates is the same as in the first embodiment, and a description thereof will be omitted.

  As shown in FIG. 10, the information processing apparatus 200 includes an image input unit 101, a sensor information acquisition unit 201, a first position recognition unit 202, a second position recognition unit 203, a shape recognition unit 103, a motion recognition unit 104, and a musical instrument recognition unit. 105, a relevance recognition unit 106, a performance information generation unit 107, and a score information generation unit 108. The image input unit 101, the shape recognition unit 103, the motion recognition unit 104, the musical instrument recognition unit 105, the relevance recognition unit 106, the performance information generation unit 107, and the score information generation unit 108 are the same as those in the first embodiment. .

  The sensor information acquisition unit 201 acquires sensor information acquired by an external sensor provided in the terminal device 10 or connected to the terminal device 10 and supplies the acquired sensor information to the second position recognition unit 203. The sensors include a microphone, a pressure sensor, a motion sensor, and the like.

  The first position recognition unit 202 recognizes the position of the player's hand that is not hidden in the input image, and is similar to the position recognition unit 102 in the first embodiment.

  Similar to the position recognition unit 102 according to the first embodiment, the first position recognition unit 202 uses a hand recognition technique for a human body called Hand Pose Detection, Hand Pose Estimation, or the like, and features such as HOG and SIFT from an input image. The three-dimensional position of the hand, which is a part of the player's body in the input image, is recognized by an extraction method, a subject recognition method by pattern recognition such as Boosting, SVM, a region extraction method by Graph Cut, or the like, or a CNN.

  The second position recognition unit 203 recognizes, using the auxiliary information, the three-dimensional position of the player's hand that is partially hidden by being occluded in the input image. The auxiliary information includes sensor information supplied from the sensor information acquisition unit 201 and the like. Examples of the sensor information include a performance sound collected by a microphone, pressure sensor information indicating a force of a hand or a finger pressing an instrument, and motion sensor information indicating a movement of a player's arm / hand / finger. Further, the auxiliary information includes position / shape / movement information of the player's arm and / or elbow recognized using the same method as the first position recognition unit 202.

  For example, based on information on the positions and shapes of the player's arms and elbows, the position of the hand (hidden hand) at the tip of the arm beyond the player's elbow is estimated and recognized in the playing area of the instrument. can do.

  Since the position information acquired by the first position recognizing unit 202 and the second position recognizing unit 203 is information indicating a three-dimensional position, for example, the (x, y, z) with a predetermined position of the input image as the origin Expressed in coordinates. Further, in a cut-out image obtained by cutting out a partial area of the input image, coordinates are represented by (u, v, d) with a predetermined position of the cut-out image as an origin. This is the same as in the first embodiment. The position information is supplied to the shape recognition unit 103 and the association recognition unit 106.

  The information processing device 200 according to the second embodiment is configured as described above.

[2-2. Processing of information processing device]
Next, a flow of processing of the information processing device 200 according to the second embodiment will be described. The flowchart of FIG. 11 corresponds to the processing for generating the partial performance information corresponding to one input image described in the first embodiment.

  First, in step S101, when an input image is input to the image input unit 101, it is determined in step S201 whether a part of the hand is hidden in the input image. This can be determined, for example, based on whether the first position recognition unit 202 has recognized the entire two hands.

  When a part of the hand is hidden, the process proceeds to step S202 (Yes in step S201), and the second position recognition unit 203 recognizes the partly hidden player's hand using the auxiliary information.

  On the other hand, if a part of the hand is not hidden, the process proceeds to step S103, where the first position recognition unit 202 recognizes the player's hand.

  Subsequent processes are the same as those in the first embodiment.

  Also, as shown in the flowchart of FIG. 12, in a process of generating composite performance information of a phrase composed of a plurality of input images and a part or all of a tune, the same as steps S201 and S202 in the flowchart of FIG. Processing is performed.

  According to the second embodiment, even if a part of the player's hand is hidden in the input image, performance information and score information can be generated in the same manner as in the first embodiment.

<3. Third Embodiment>
[3-1. Configuration of Information Processing Apparatus]
Next, a third embodiment of the present technology will be described. In the third embodiment, as shown in FIG. 13, performance information is generated when a part of a musical instrument is hidden or not shown in an input image. In FIG. 13, only a part of the keyboard of the piano which is the musical instrument is shown, and a part of the keyboard exists outside the angle of view of the input image. The configuration of the terminal device 10 on which the information processing device 300 operates is the same as in the first embodiment, and a description thereof will be omitted.

  As shown in FIG. 14, the information processing apparatus 300 includes an image input unit 101, a sensor information acquisition unit 301, a position recognition unit 102, a shape recognition unit 103, a motion recognition unit 104, a musical instrument recognition unit 105, a relevance recognition unit 106, It comprises a performance information generator 107 and a score information generator 108. The image input unit 101, the position recognition unit 102, the shape recognition unit 103, the motion recognition unit 104, the performance information generation unit 107, and the score information generation unit 108 are the same as those in the first embodiment.

  The sensor information acquiring unit 201 acquires sensor information acquired by an external sensor provided in the terminal device 10 or connected to the terminal device 10 and supplies the acquired sensor information to the performance information generating unit 107. The sensors include a microphone, a pressure sensor, a motion sensor, and the like.

  The musical instrument recognizing unit 105 recognizes a musical instrument in an input image and a performance area (performance area) in which the player's hand touches the musical instrument in the input image using techniques such as CNN, pattern matching, and template matching. is there. Therefore, for example, when only a part of the musical instrument matches the template in the template matching, it is determined that the recognized musical instrument has a part hidden or not reflected. The instrument recognition information is supplied to the association recognition unit 106 together with information indicating that only a part of the instrument is reflected in the input image.

  The relevance recognition unit 106 recognizes the relevance between the position of the player's hand and the playing area of the musical instrument using techniques such as CNN and pattern matching. The relevance is a contact position indicating where the player's hand is in contact with the playing area of the instrument. The relevance is the direction of movement of the player's hand with respect to the playing area of the instrument. When only a part of the musical instrument is shown in the input image, the relevance recognizing unit 106 sets the hand position information, the hand shape information, and the substantially horizontal direction with respect to the direction in which the keyboard, which is the playing area of the musical instrument (eg, piano), is arranged. By recognizing the performance area where the finger is in contact with the angle of the arm / elbow opening and the movement of the arm, the relevance between the player's finger and the performance area of the instrument is recognized. The relevancy information is supplied to the performance information generation unit 107.

  The performance information generation unit 107 estimates a keyboard with which a finger is in contact using sensor information as auxiliary information. Examples of the sensor information include a performance sound collected by a microphone, pressure sensor information indicating a force of a hand or a finger pressing an instrument, and motion sensor information indicating a movement of a player's arm / hand / finger. Further, when there is an input image in which the entire instrument is captured in a plurality of input images, the relevance recognition unit 106 estimates the keyboard with which a finger is in contact from the input image and arm and hand movement information, thereby performing To estimate the relevance between the finger and the playing area of the instrument.

  By estimating the keyboard with which the finger is in contact, the first performance element, the second performance element, and the third performance element can be generated from the estimation result in the same manner as in the first embodiment.

  The information processing device 300 according to the third embodiment is configured as described above.

[3-2. Processing of information processing device]
Next, a flow of processing of the information processing device 300 according to the third embodiment will be described. The flowchart of FIG. 15 corresponds to the processing for generating the performance information and the musical score information corresponding to one input image described in the first embodiment.

  Steps S101 to S105 are the same as the processing in the first embodiment.

  In step S301, the relevance recognition unit 106 determines whether or not the entire musical performance area of the musical instrument is included in the input image. If the entire musical play area is included, the process proceeds to step S106 (Yes in step S301). Then, the processing of steps S106 to S112 is performed in the same manner as in the first embodiment.

  On the other hand, if the entire performance area of the musical instrument is not shown in the input image, the process proceeds to step S302 (No in step S301). Then, in step S302, the relevance is estimated by the relevance recognition unit 106 using the hand position information, the sensor information, and the like.

  Thereafter, the processing of steps S106 to S112 is performed in the same manner as in the first embodiment, and partial performance information is generated and output.

  Also, as shown in the flowchart of FIG. 16, in a process of generating composite performance information of a phrase composed of a plurality of input images and a part or all of a song, the same processing as in steps S301 and S302 in the flowchart of FIG. Processing is performed.

  According to the third embodiment, performance information and musical score information can be generated similarly to the first embodiment even if a part of the musical instrument is not shown in the input image.

<4. Modification>
Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of the present technology are possible.

  In the embodiment, it has been described that the performance information and the score information can be generated from a plurality of continuous still images or a plurality of frame images constituting a moving image without the sound of the performance, but the present technology excludes the use of the sound. Not something. Sound information may be used as auxiliary information when generating performance information and musical score information, or sound information may be used when checking the accuracy of the generated performance information and musical score information. For example, voice recognition processing is performed on the audio of the input video to recognize the scale based on the frequency of the sound, and to recognize whether the sound is strong or weak based on the volume.

  By combining the second embodiment and the third embodiment, it is possible to generate performance information even when a part of a player's hand and a part of a performance area of a musical instrument are not reflected in an input image. it can.

  The present technology is not limited to the piano, guitar, and drum described in the embodiment, and can be used for playing musical instruments such as a xylophone, a metallophone, and a percussion.

  In the embodiment, the description has been given mainly of the piano playing method such as pressing and striking, and the guitar playing method such as stroking a hand and striking a nail, but other playing methods such as pulling and playing The performance information may be generated by recognizing a performance operation such as a performance operation. The performance information may be generated based on any operation of the performance of the musical instrument that can be recognized from the input image.

  In the third embodiment, a part of the musical instrument not shown in the input image may be estimated, and the performance information generation unit 107 may generate the performance information based on the estimation result.

The present technology can also have the following configurations.
(1)
A position recognition unit that recognizes a position of a body part of the player from the input image;
An instrument recognition unit that recognizes an instrument from the input image;
An information processing apparatus, comprising: a performance information generating unit configured to generate performance information indicating performance of the musical instrument by the player based on a relationship between the position of the part and the musical instrument.
(2)
A shape recognition unit that recognizes the shape of the part recognized by the position recognition unit,
The information processing device according to (1), wherein the performance information generation unit generates the performance information based on a relationship between the shape of the part and the musical instrument.
(3)
A movement recognition unit that recognizes the movement of the part recognized by the position recognition unit,
The information processing device according to (1) or (2), wherein the performance information generation unit generates the performance information based on the association between the movement of the part and the musical instrument.
(4)
The information processing apparatus according to any one of (1) to (3), wherein the performance information includes a first performance element corresponding to a state in which the player is playing the musical instrument.
(5)
The information processing device according to (4), wherein the first performance element includes a scale played by the player.
(6)
2. The information processing apparatus according to claim 1, wherein the performance information includes a second performance element corresponding to a state in which the player does not play the musical instrument. 3.
(7)
The information processing device according to (6), wherein the second performance element includes a length of a rest that is not performed by the player.
(8)
The information processing apparatus according to any one of (1) to (7), wherein the performance information includes a third performance element that is an element spanning the plurality of input images.
(9)
The information processing device according to (8), wherein the third performance element includes a tempo of a song played by the player.
(10)
The information processing apparatus according to any one of (1) to (9), wherein the performance information generation unit generates the performance information corresponding to one input image.
(11)
The information processing apparatus according to any one of (1) to (9), wherein the performance information generation unit generates performance information corresponding to a part or all of the performance of the musical instrument constituted by the plurality of input images.
(12)
The information processing apparatus according to any one of (1) to (11), wherein the association is a contact position of the part with the musical instrument.
(13)
The information processing apparatus according to any one of (1) to (12), wherein the association is a direction of movement of the part with respect to the musical instrument.
(14)
The information processing apparatus according to any one of (1) to (13), wherein the part is a hand of the player.
(15)
The information processing apparatus according to any one of (1) to (14), further comprising a score information generating unit configured to generate score information from the performance information.
(16)
When a part of the part is not shown in the input image, the position of a part of the part is estimated, and the performance information generation unit generates the performance information based on the estimation result (1) to (15). An information processing device according to any one of the above.
(17)
When a part of the musical instrument is not shown in the input image, the performance information generation unit estimates the association between the part and the part of the musical instrument, and generates the performance information based on the estimation result ( The information processing apparatus according to any one of (1) to (16).
(18)
The information processing apparatus according to any one of (1) to (17), wherein the performance information generation unit generates the performance information using sound information as auxiliary information.
(19)
Recognize the position of the performer's body from the input image,
Recognizing a musical instrument from the input image,
An information processing method for generating performance information indicating a performance of the musical instrument by the player based on a relationship between the position of the part and the musical instrument.
(20)
Recognize the position of the performer's body from the input image,
Recognizing a musical instrument from the input image,
An information processing program for causing a computer to execute an information processing method for generating performance information indicating a performance of the musical instrument by the player based on a relationship between the position of the part and the musical instrument.

100, 200, 300 Information processing device 102 Position recognition unit 103 Shape recognition unit 104 Motion recognition unit 105 Instrument recognition unit 107 Performance information generation unit 108・ Score information generation unit 202: first position recognition unit 203: second position recognition unit

Claims (20)

A position recognition unit that recognizes a position of a body part of the player from the input image;
An instrument recognition unit that recognizes an instrument from the input image;
An information processing apparatus, comprising: a performance information generating unit configured to generate performance information indicating performance of the musical instrument by the player based on a relationship between the position of the part and the musical instrument. A shape recognition unit that recognizes the shape of the part recognized by the position recognition unit,
The information processing apparatus according to claim 1, wherein the performance information generating unit generates the performance information based on a relationship between the shape of the part and the musical instrument. A movement recognition unit that recognizes the movement of the part recognized by the position recognition unit,
The information processing apparatus according to claim 1, wherein the performance information generation unit generates the performance information based on a relationship between the movement of the part and the musical instrument. The information processing apparatus according to claim 1, wherein the performance information includes a first performance element corresponding to a state in which the player is playing the musical instrument. The information processing apparatus according to claim 4, wherein the first performance element includes a scale played by the player. The information processing apparatus according to claim 1, wherein the performance information includes a second performance element corresponding to a state in which the player does not play the musical instrument. The information processing apparatus according to claim 6, wherein the second performance element includes a length of a rest that has not been performed by the player. The information processing apparatus according to claim 1, wherein the performance information includes a third performance element that is an element spanning the plurality of input images. 9. The information processing apparatus according to claim 8, wherein the third performance element includes a tempo of a music piece played by the player. The information processing apparatus according to claim 1, wherein the performance information generation unit generates the performance information corresponding to one of the input images. The information processing apparatus according to claim 1, wherein the performance information generation unit generates performance information corresponding to a part or all of the performance of the musical instrument composed of the plurality of input images. The information processing apparatus according to claim 1, wherein the association is a contact position of the part with the musical instrument. The information processing apparatus according to claim 1, wherein the association is a direction of movement of the part with respect to the musical instrument. The information processing apparatus according to claim 1, wherein the part is a hand of the player. The information processing apparatus according to claim 1, further comprising a musical score information generating unit configured to generate musical score information from the performance information. The information processing according to claim 1, wherein when a part of the part is not shown in the input image, a position of the part is estimated, and the performance information generating unit generates the performance information based on the estimation result. apparatus. When a part of the musical instrument is not shown in the input image, the performance information generation unit estimates the association between the part and the part of the musical instrument, and generates the performance information based on the estimation result. Item 2. The information processing device according to item 1. The information processing apparatus according to claim 1, wherein the performance information generation unit generates the performance information using sound information as auxiliary information. Recognize the position of the performer's body from the input image,
Recognizing a musical instrument from the input image,
An information processing method for generating performance information indicating a performance of the musical instrument by the player based on a relationship between the position of the part and the musical instrument. Recognize the position of the performer's body from the input image,
Recognizing a musical instrument from the input image,
An information processing program for causing a computer to execute an information processing method for generating performance information indicating a performance of the musical instrument by the player based on a relationship between the position of the part and the musical instrument.

Images