JP2002041038A - Virtual musical instrument playing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual musical instrument playing device which compounds one's self shot by a video camera with an image of a virtual instrument on a TV screen (display), and can be played with looking at the image. SOLUTION: The image of a player is taken in with a video camera part 2a and 2b, and the position of the player's movement for playing an instrument is detected by a three-dimensions-position-detector 4. An operating device 5 compounds an image containing the virtual musical instrument which is output from a storage 8 with an image of the player from the three-dimensions-position- detector 4, and displays it on a display 7, besides, the player operates it with looking at the position of the virtual musical instrument on the display 7, and a playing information of the musical instrument is made up from an information of fingers when silhouettes of the instrument and the player on the two dimensions are met. A sound based on the playing information, which is sent from the operating device 5 by a musical sound generating device 6, is output from a speaker 3.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual musical instrument playing apparatus capable of playing a virtual musical instrument on a TV monitor.

[0002]

2. Description of the Related Art If one can play a musical instrument not as an actual musical instrument but as a virtual instrument as one of the entertainment equipment, it is possible to easily enjoy the performance.

[0003]

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and has as its object to image a virtual musical instrument on a TV screen (display) by photographing oneself (or a doll on behalf of oneself) photographed by a video camera. It is an object of the present invention to provide a virtual musical instrument playing apparatus that can perform a performance by moving while watching the video.

[0004]

In order to achieve the above object, a virtual musical instrument playing apparatus according to the present invention comprises a camera section for capturing an image of a player, and an operating position for the player to operate the musical instrument from the captured image. A three-dimensional position detector for detecting an image, a storage device for storing at least one image including a virtual musical instrument,
A display device, an image including the virtual instrument output from the storage device and an image of the player from the three-dimensional position detector are combined and displayed on the display device, and the player The operation is performed while observing the position, and an arithmetic unit for producing musical instrument performance information from positional information of the hand when the virtual musical instrument and the performer's two-dimensional contour come into contact with each other, according to the performance information sent from the arithmetic unit. And a tone generating device for generating a tone. In the present invention,
The image of the player displayed on the display device is displayed as a left-right inverted image. Further, according to the present invention, when the player touches the operation unit of the virtual musical instrument, the sound is generated, and the volume change, the timbre change, and the pitch change are made to correspond to the movement amount of the player's hand from the position at which the sound is started. It is configured to change.

[0005]

According to the above arrangement, the virtual musical instrument can be played on the TV monitor.

[0006]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of a virtual musical instrument playing apparatus according to the present invention. The virtual musical instrument playing device includes video cameras 2a and 2b, a speaker 3, and a virtual musical instrument playing device unit 9. The virtual musical instrument playing device section 9 includes a three-dimensional position detector 4, a computing device 5, a musical sound generating device 6, a display device 7, and a storage device 8.

The video cameras 2a and 2b three-dimensionally capture an image of a player (a substitute doll may be used (hereinafter referred to as a "player"). The three-dimensional position detector 4 extracts the contour of the player image captured from the video cameras 2a and 2b, and detects the hand, head and waist position information of the player. The storage device 8 stores a plurality of images including virtual musical instruments, and the computing device 5 combines and displays the image read from the storage device 8 and the inverted image of the performer 1 captured from the video cameras 2a and 2b. Output to the device 7. Then, the contours of the performer's images captured from the video cameras 2a and 2b are extracted, and the performance of the musical instrument is obtained from the positional information of the hand, the head, and the abdomen when the virtual musical instrument and the performer's two-dimensional contour come into contact with each other. Information is created and sent to the tone generator 6. The tone generator 6 generates a tone based on the performance information and outputs the tone from the speaker 3.

[0008] There are two types of operating means for the virtual musical instrument. Basically, there are two types of detection methods, one is detection of an operation that sounds a sound, and the other is detection of an operation that changes the whole sound in various ways. When sound is generated, the contour of the image of the hand of the player or the like is adjusted based on the three-dimensional position detected at the moment when the contour of the musical instrument located above the player in the normal image is touched, and the moving speed of the hand. Pronounced at the loudest volume. After that, how much the hand position has moved from the contact position is output as a change factor of the timbre (one of volume, pitch, or sound quality) of the sound. Further, when the contour of the player's hand deviates from the contour of the musical instrument, the sound generation ends.

To change the overall sound, the outline of the image of the player's hand touches the outline of the operation buttons located on the left and right sides of the player on the normal screen, and moves in a certain direction after the contact (for example, The function starts when it is 10 cm forward from the point of view of the player), and each movement distance in the two remaining three-dimensional directions from that point is assigned to the entire timbre change. The above-mentioned sound generation, individual timbre changes, and overall timbre changes can be substituted for a keyboard such as a synthesizer and a ribbon controller or wheel. Here, the change of each sound means to change the sound that is sounded by one trigger, and the change of the whole sound means that two or more sounds are sounded and changed together.

FIG. 2 is a flowchart for explaining the operation of the virtual musical instrument playing apparatus according to the present invention. The musical instrument is displayed on the display device 7 by the start of the virtual musical instrument performance (steps (hereinafter referred to as "S") 201, S202). The image of the player 1 is captured from the video cameras 2a, 2b, and the arithmetic unit 5 includes the musical instrument. And the left-right inverted image of the performer are synthesized and displayed on the display device 7 (S20).
3).

The three-dimensional position detector 4 detects the three-dimensional position of the movement of the player 1 and the like, and the contour (two-dimensional) of the player.
Is detected (S204, S205). The arithmetic unit 5
It is determined whether the virtual instrument and the player have contacted each other (S20).
6). If not, the process returns to S204. When touched, performance information is generated and sent to the tone generator 6.
The tone generator 6 generates a tone based on the performance information and outputs the tone from the speaker 3 (S207). Then, it is determined whether or not the performance has ended (S208). If not, the process returns to S204. When the performance ends, the virtual musical instrument is closed (S209).

FIG. 3 is a schematic perspective view showing a performance example of the virtual musical instrument performance apparatus according to the present invention. A player 12 is standing in front of a large TV screen 15, and video cameras 13 a and 13 b are installed on both sides of the large TV screen 15, respectively. The performer 12 is displayed on the large TV screen 15 in a mirror-reversed manner, like a mirror, and a virtual musical instrument is also synthesized and displayed.

In this example, the virtual musical instruments are arranged as upper six buttons 11a to 11f and one lower right button 10. When the hand of the player 12 shown on the screen touches any one of the six buttons at the top of the screen on the screen, sound generation starts. The higher the speed of touching the button, the louder the volume. When the player's hand is released from the button, the sound generation stops. Depending on the sound to be pronounced, it is possible to prevent the sound from being stopped even when the hand is released from the button, as in the case of a percussion instrument. If the user moves the hand while touching the button to produce sound, the volume, pitch, and sound quality are changed according to the amount of movement.

When the player 12 touches the lower right button 10 and pushes his / her hand 10 cm forward, for example, the controller can be used. After functioning, the controller can change the volume, pitch, sound quality, and the like of all the sounds being generated according to the movement of the hand, such as left and right. For example, moving the hand to the left and right can raise or lower the pitch of all sounds. In addition, the volume and the sound quality can be changed by moving up and down.

[0015]

As described above, according to the present invention, it is possible to play a new sense by operating a sound (sound source) without touching an actual musical instrument, thereby realizing an unprecedented virtual musical instrument playing apparatus. Is what you do. Further, the present invention can be used not only as a musical instrument but also as an input device for a game or the like. In addition to karaoke, the present invention can be applied to playing with the virtual musical instrument together with the accompaniment and adding a sound effect.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a virtual musical instrument playing device according to the present invention.

FIG. 2 is a flowchart illustrating the operation of the virtual musical instrument playing apparatus according to the present invention.

FIG. 3 is a schematic perspective view showing a performance example of the virtual musical instrument performance device according to the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Performer etc. 2a, 2b, 13a, 13b ... Camera 3 ... Speaker 4 ... Three-dimensional position detector 5 ... Arithmetic device 6 ... Musical sound generator 9, 14 ... Virtual musical instrument performance device part 10, 11a-11f ... Operation part 12 ... Performer 15 ... Large TV screen

Claims (3)

[Claims] 1. A camera unit for capturing an image of a player, a three-dimensional position detector for detecting an operating position for a player to operate a musical instrument from the captured image, and one or more images including virtual musical instruments. A storage device, a display device, and an image including the virtual musical instrument output from the storage device and an image of the player from the three-dimensional position detector are combined and displayed on the display device. An operation device which operates while observing the position of the virtual musical instrument, and generates performance information of the musical instrument from positional information of the hand when the virtual musical instrument and the two-dimensional contour of the player come into contact with each other. A musical instrument generating apparatus for generating a musical tone according to performance information, comprising: 2. The virtual musical instrument playing device according to claim 1, wherein the image of the player displayed on the display device is a left-right inverted image. 3. When a player touches the operation unit of the virtual musical instrument, a sound is generated, a volume change, a timbre change, and a pitch change are performed.
2. The virtual musical instrument playing apparatus according to claim 1, wherein the virtual musical instrument playing apparatus is configured to change from a position at which the sound is started to correspond to a moving amount of the player's hand.