JP7070538B2 - Programs, methods, electronic devices, and performance data display systems - Google Patents

Description

Embodiments relate to programs, methods, electronic devices, and performance data display systems.

For example, an image object such as a dancer (dansa) is moved in synchronization with the performance of music such as MIDI (Musical Computer Interface) music, and it is integrated with this not only according to the musical composition but also according to the progress of the musical sound. A computer graphics moving image generation system capable of generating a changing moving image is known (see Patent Document 1).

Japanese Unexamined Patent Publication No. 11-224084

In the computer graphics moving image generation system disclosed in Patent Document 1, there is a problem that the expressive power is poor because the character showing the dancer is simply dynamically made to appear during the performance.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide, for example, a program, a method, an electronic device, and a performance data display system having excellent visual expressiveness of performance.

According to the program according to the embodiment, the computer of the information processing apparatus is made to display the first image data according to the information that can be determined based on the acquired first performance data, and is acquired after the acquisition of the first performance data. It is determined whether or not the second performance data to be performed satisfies the set condition, and when it is determined that the set condition is satisfied, the third performance data acquired after the acquisition of the second performance data is performed. Even if the information that can be determined based on the above is the same as the information that can be determined based on the first performance data, the second image data in a mode different from that of the first image data is displayed.

According to the present invention, for example, it is possible to provide a program, a method, an electronic device, and a performance data display system having excellent visual expressiveness of a performance.

FIG. 1 is a diagram schematically illustrating an example of a performance data display system including a program according to an application example. FIG. 2 is a diagram showing an outline of a performance data display system according to an embodiment. FIG. 3 is a diagram showing hardware of a control system for an electronic musical instrument according to an embodiment. FIG. 4 is a diagram showing the hardware of the information processing apparatus according to the embodiment. FIG. 5 is a diagram showing an example of a table stored in a ROM table. FIG. 6 is a diagram showing a display example of an image (character). FIG. 7 is a diagram showing an example of an image (character) trajectory pattern in a real-time image displayed on a display unit of a display device. FIG. 8 is a diagram showing a spiral winding which is an example of the trajectory pattern generated by the image determination unit. FIG. 9 is a flowchart showing the operation of the information processing apparatus. FIG. 10 shows a musical score played by the user. FIG. 11 is a flowchart showing an example of a method of registering “set conditions” for the control unit to determine in step S1004 of FIG. FIG. 12 is a flowchart showing a specific example of the determination method of the “set condition” in step S1004 of FIG. FIG. 13 shows a real-time image when the user is playing the second phrase in FIG. FIG. 14 shows a real-time image when the user is playing the third phrase in FIG. FIG. 15 shows a post-performance image after the user has finished playing the song shown in FIG.

Hereinafter, embodiments according to one aspect of the present invention (hereinafter, also referred to as “embodiments”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. The data appearing in the embodiment are described in natural language, but more specifically, they are specified in a pseudo language, a command, a parameter, a machine language, etc. that can be recognized by a computer.

<0> Application Example First, an example of a situation in which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates an example of a performance data display system including a program according to an application example. The performance data display system is supposed to stimulate the user's motivation to practice an electronic musical instrument by drawing an image (picture) in real time according to the performance (music) of the user (performer).

<0-1> Outline of application example
Before explaining the configuration of the performance data display system, the outline of the performance data display system will be described. The performance data display system analyzes performance data received from an electronic musical instrument or the like that can output the user's performance as performance data (for example, MIDI data), and converts the user's performance into an image. For example, the performance data does not include the note name information and the chord type information, and the note name information and the chord type information can be determined from the note number information included in the performance data.

<0-2> Configuration of Application Example The configuration of the application example will be described with reference to FIG. FIG. 1 is a diagram showing an example of the configuration of a performance data display system according to an application example.

As shown in FIG. 1, the performance data display system includes an electronic musical instrument, an information processing device, and a display device.

The electronic musical instrument converts the user's performance into performance data (for example, MIDI data). Then, the electronic musical instrument outputs the performance data to the information processing device.

More specifically, the electronic musical instrument generates the first performance data according to the first performance operation by the user, and outputs the generated first performance data. Further, the electronic musical instrument generates the second performance data according to the second performance operation by the user, and outputs the generated second performance data. For example, the second performance operation is an operation following the first performance operation. The electronic musical instrument generates the third performance data according to the third performance operation by the user, and outputs the generated third performance data. For example, the third performance operation is an operation following the second performance operation. The electronic musical instrument is, for example, a keyboard, an electronic piano, an electric guitar, an electronic violin, or the like. However, the electronic musical instrument is not limited to the above-mentioned one, and may be appropriately selected according to the embodiment.

The information processing device analyzes the received performance data and converts it into image data. More specifically, the information processing apparatus outputs the first image (character) data according to the information (for example, note name, chord type, etc.) that can be determined based on the acquired first performance data. Further, the information processing apparatus determines whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set condition, and if it is determined that the set condition is satisfied, the second performance data is obtained. 2 Even if the information that can be determined based on the third performance data acquired after the acquisition of the performance data (for example, sound name, chord type, etc.) is the same as the information that can be determined based on the first performance data, the first The second image (character) data having a mode different from that of the first image data is output. The information processing device is, for example, a PC (personal computer). Note that "setting" or "registration" basically has the same meaning as "memory". Further, a specific example of the "set condition" will be described later.

The display device displays an image based on the user's performance. More specifically, the display device displays the image data received from the information processing device on the display unit.

<0-3> Operation of application example Next, an example of operation of the performance data display system according to the application example will be described.

When the user starts playing with an electronic musical instrument, the information processing device performs music analysis on the performance. Specifically, the information processing apparatus has an image (for example, a beetle) included in the first image group (for example, a character group including a plurality of insect characters) according to a sound name (or chord) included in the performance data. The character) is determined and drawn in real time along the trajectory according to the set first condition (for example, display according to the code progress). As a result, the image (character) included in the first image group is displayed on the display device in real time. The "real time" here does not necessarily mean that the timing of the performance and the timing of displaying the image (character) are the same. Further, an image in which an image (character) is displayed according to the performance may be described as, for example, a real-time image. That is, a plurality of images (characters) are drawn in the real-time image. The drawing method of the real-time image will be described later.

If the user plays a performance that meets the conditions set (stored or registered) in the information processing device while the user is playing an electronic musical instrument, the information processing device sets the first image group (for example, a bug character). Switch from the second image group (for example, a character group containing a plurality of animal characters) from the second image group (for example, a character group containing a plurality of animal characters), and according to the sound name (or chord) included in the performance data, the second image group (for example, a plurality of animal characters). An image (for example, a giraffe character) included in the included character group) is determined and drawn in real time along a trajectory according to a set first condition (for example, display according to code progression). As a result, the image (character) included in the second image group is displayed on the display device in real time. It should be noted that switching the image group is also described as "changing drawing settings".

Further, after the user finishes playing the electronic musical instrument, the information processing device causes the display device to display the image data according to the set second condition (for example, swirl display). In addition, such an image may be described as, for example, a post-performance image. The post-performance image is an image in which an image (character) analyzed from the beginning to the end of the user's performance is drawn according to a second condition (for example, a spiral display).

<0-4> Effect of application example As described above, according to the performance data display system according to the application example, the display is performed by satisfying a preset condition while the user is playing an electronic musical instrument. You can change the group of images you want. In other words, according to the performance data display system according to the application example, it is possible to appropriately set the image-related settings (drawing settings) during the performance of the user without performing a peculiar operation unrelated to the performance. Therefore, the performance data display system can generate a highly expressive image while the user concentrates on the performance.

<1> Embodiment <1-1> Configuration <1-1-1> Performance data display system The performance data display system according to the embodiment will be described below with reference to FIG. 2.

FIG. 2 is a diagram showing an outline of a performance data display system according to an embodiment.

As shown in FIG. 2, the performance data display system includes, for example, an information processing device (electronic device) 1, an electronic musical instrument 2, a speaker 3, and a display device 4.

<1-1-2> Electronic Musical Instrument Hereinafter, the electronic musical instrument according to the embodiment will be described with reference to FIGS. 2 and 3. FIG. 3 is a diagram showing hardware of a control system for an electronic musical instrument according to an embodiment.

As shown in FIGS. 2 and 3, the electronic musical instrument 2 to which the information processing device 1 of the present embodiment is connected is, for example, an electronic keyboard instrument such as an electronic piano, a synthesizer, or an electronic organ.

The electronic musical instrument 2 includes a plurality of keys 11, a display unit 20, an operation unit 30, and a sounding unit 40.

Further, as shown in FIG. 3, the electronic musical instrument 2 includes a key pressing detection unit 50, a storage unit 60, a control unit 70, and a communication unit 80.

The key 11 is an operator for designating a pitch by the performer, and when the performer presses and releases the key 11, the electronic musical instrument 2 produces and mutes the sound corresponding to the designated pitch. I do.

The display unit 20 has, for example, a liquid crystal monitor with a touch panel, and is a unit that displays a message or the like accompanying the operation of the operation unit 30 of the performer.

In this embodiment, since the display unit 20 has a touch panel function, the display unit 20 can play a part of the operation unit 30.

The operation unit 30 has operation buttons for performing various settings and the like, and is a part for performing various setting operations such as volume adjustment and the like.

The sounding unit 40 is a part that outputs sound, and is an HP jack 44 (headphone) into which a plug of an SP amplifier 41 (speaker amplifier), a speaker 42, an HP amplifier 43 (headphone amplifier), and a speaker 3 (or headphones) is inserted. It has a jack) and an HP jack insertion detection unit 45 that detects that the plug of the speaker 3 (or headphones) is inserted into the HP jack 44.

When the plug of the speaker 3 (or headphones) is inserted into the HP jack 44, the HP jack insertion detection unit 45 detects the insertion, the sound is output to the HP jack side, and conversely, the HP jack insertion detection is performed. When the unit 45 does not detect the insertion of the plug, the sound is output to the speaker 42 side.

The sounding unit 40 may be used instead of the speaker 3.

The key press detection unit 50 is a part that detects the key press and release of the key 11. For example, a rubber switch is provided under each key 11, and the key press and release are detected by the rubber switch. To.

The storage unit 60 includes a ROM 61 which is a read-only storage unit and a RAM 62 which is a read / write storage unit.

The ROM 61 stores, for example, a control program (for example, a program for a lesson mode program) executed by the control unit 70, various data tables, and the like.

The RAM 62 is a storage area for storing music data, music data, and other data corresponding to each key 11, and is also a temporary storage area for developing data generated by the control unit 70 and a control program during performance. Also works as.

The control unit 70 is a part that controls the entire electronic musical instrument 2. The control unit 70 is, for example, an LSI (Large Scale Integration).

Then, the control unit 70 controls, for example, to make the musical sound sound from the sounding unit 40 in response to the key pressed by the key 11, or to mute the musical sound sounded from the sounding unit 40 in response to the release of the key 11. conduct.

The communication unit 80 includes a wireless unit and a wired unit for communicating with an external device. In the present embodiment, the information processing device 1 is connected by wire, and as will be described later, information processing is performed. Data is transmitted to the device 1.

However, the communication unit 80 is not limited to the one that performs only the transmission function, and may also have the reception function.

Then, each unit (display unit 20, operation unit 30, sound generation unit 40, key press detection unit 50, storage unit 60, control unit 70, and communication unit 80) described above are connected to each other so as to be communicable by the bus 100. Therefore, necessary data can be exchanged between each department.

<1-1-3> Information Processing Device Next, the information processing device will be described with reference to FIGS. 2 and 4. FIG. 4 is a diagram showing the hardware of the information processing apparatus according to the embodiment.

As shown in FIG. 2, the information processing apparatus 1 is a PC, and an application for displaying an image as described later on the display apparatus 4 is installed in the information processing apparatus 1.

However, the information processing device 1 does not have to be limited to a PC, and may be a tablet-type mobile terminal or the like.

Further, the information processing device 1 may include a sequencer or the like that receives a MIDI code from the electronic musical instrument 2 and reproduces the music data.

As shown in FIG. 4, the information processing apparatus 1 mainly includes an operation unit 101, a display unit 102, a communication unit 103, a sound output unit 104, a control unit 105, and a storage unit 106. There is.

The operation unit 101 includes, for example, switches such as a power switch for turning on / off the power of the information processing device 1.

The display unit 102 has a liquid crystal monitor and can display an image. The display unit 102 may have a touch panel function. In that case, the display unit 102 can bear one end of the operation unit 101. The display unit 102 may be used instead of the display device 4.

The communication unit 103 includes a wireless unit and a wired unit for communicating with other devices and the like. The communication unit 103 is connected by wire to, for example, the communication unit of the electronic musical instrument 2, and receives data from the electronic musical instrument 2. Further, the communication unit 103 is connected by wire to, for example, the communication unit of the display device 4, and transmits data to the display device 4.

The sound output unit 104 is a part that outputs sound, and has a speaker, an earphone jack, and the like.

The storage unit 106 includes a RAM (Random Access Memory) 1061 which is a read / write storage unit which is a read-only storage unit, and a ROM (Read Only Memory) 1062.

The RAM 1061 includes a volatile memory, is a storage area for storing data necessary for operating a control program, data generated by the control unit 105, and data sent from the electronic instrument 2 (sound pitch information). It also functions as a temporary storage area for expanding the application (including performance information).

The ROM 1062 is a non-volatile semiconductor memory such as a flash memory, EPROM (Erasable Project Read Only Memory), and EEPROM (registered trademark) (Electrically Erasable Project ROM), and serves as a so-called secondary storage device or auxiliary storage device. ..

ROM 1062 stores program 10621 and table 10622. The program 10621 is used by the control unit 105 to perform various processes. Table 10622 is a table in which performance information and images are linked.

The control unit 105 is a part that controls the entire information processing apparatus 1. The control unit 105 includes, for example, a CPU (Central Processing Unit). The control unit 105 collectively controls the information processing apparatus 1 by reading the program 10621 stored in the ROM 1062 of the storage unit 106 and using the RAM 1061 of the storage unit 106 as a work area.

Next, the functional configuration of the control unit 105 of the information processing apparatus 1 according to the embodiment will be described. By reading and executing the program 10621 stored in the ROM 1062, the control unit 105 reads out and executes the performance determination unit 1051, the image determination unit 1052, the image information output unit 1053, the condition determination unit 1054, and the performance end determination unit 1055. And the condition registration unit 1056. The performance determination unit 1051, the image determination unit 1052, the image information output unit 1053, the condition determination unit 1054, the performance end determination unit 1055, and the condition registration unit 1056 are not based on the program but on the circuit configuration. It may be realized.

The performance determination unit 1051 has the tonality of the song (for example, 24 types from C major to B minor) and the sound name (for example, 24 types) based on the performance information (performance data) received from the electronic musical instrument 2 via the communication unit 103. Do, re, mi ...), chord type (for example, Major, Mimor, Sus4, Aug, Dim, 7th, etc.), velocity value (numerical value indicating the strength of the sound in the MIDI standard), note length, syncopation, chord function, and Determine the chord progression. Any method may be used for determining the tonality, note name, chord type, velocity value, note length, syncopation, chord function, and chord progression of the song.

Each time the image determination unit 1052 receives performance information from the electronic musical instrument 2, any image data (character) included in the image group (character group) is selected based on the information determined by the performance determination unit 1051. decide.

Here, with reference to FIG. 5, the relationship between the information determined by the performance determination unit 1051 and the image data will be described. FIG. 5 is a diagram showing an example of a table stored in the table 10622 of the ROM 1062.

As shown in FIG. 5, in the table 10622, for example, a first image (character) in which an image (character) is set corresponding to a note name (here, an example of do, re, mi, fa, so, la, shi) is set. The group and the second image group are stored. The method of selecting the first image group and the second image group will be described later. Further, the table 10622 is not limited to this, and other note names (for example, do # etc.) and images corresponding to the chord type may be stored. For example, in the first image group and the second image group, images having the same note name but different modes are assigned.

For example, if the image determination unit 1052 determines that the note name included in the received performance information is "do", the image determination unit 1052 determines the image (character) of the "butterfly" in the first image group of the table 10622.

Here, a method of displaying an image (character) will be described with reference to FIG. FIG. 6 is a diagram showing a display example of an image (character).

As shown in FIG. 6, the image (character) stored in the table 10622 is, for example, a basic form. The image determination unit 1052 determines the size of the image (character) to be displayed on the display device 4 based on the velocity value included in the performance information. The size of the basic shape is determined to be height HB and width WB. Then, for example, by changing the percentages of the height HB and the width WB based on the velocity value, the height HC and the width WC are changed. Further, the image determination unit 1052 may randomly rotate the basic shape of the image (character) by R degree, invert it left and right, or change the color. Although not described in detail here, the image determination unit 1052 may appropriately change the size, rotation, inversion, or color based on the information not shown here.

When displaying a real-time image, the image determination unit 1052 determines a trajectory pattern (first condition) for arranging an image (character) on the display unit of the display device 4 based on performance information (for example, chord progression). The orbital pattern (first condition) is stored in, for example, ROM 1062. Here, an example of an orbital pattern when displaying an image (character) will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of an image (character) trajectory pattern in a real-time image displayed on the display unit 401 of the display device 4.

As shown in FIG. 7, as an example of the trajectory pattern, in the upper half of the real-time image (see 1st (U)), the images (characters) are sequentially arranged on the virtual arrows along the arrow A1, and the arrows. Following A1, images (characters) are sequentially arranged on virtual arrows along the arrow A2 in the lower half of the real-time image (see 1st (L)). Each image (character) is arranged so that at least a part of each image (character) overlaps on the virtual arrows (A1, A2) of the trajectory pattern.

The image determination unit 1052 generates a trajectory pattern (second condition) of the post-performance image to be displayed on the display device 4 after the user's performance. The image determination unit 1052 generates a trajectory pattern (second condition) for arranging an image (character) associated with each performance information. The image determination unit 1052 determines the length of the performance trajectory pattern (second condition) based on the performance time calculated by the performance determination unit 1051. For example, the length of the orbital pattern (second condition) when the playing time is the first playing time length and the length of the orbital pattern (second condition) when the playing time is the second playing time length. When the first playing time length is shorter than the second playing time length, the first length is shorter than the second playing time length.

An example of the trajectory pattern (second condition) generated by the image determination unit 1052 will be described with reference to FIG. FIG. 8 is a diagram showing a spiral winding which is an example of the trajectory pattern (second condition) generated by the image determination unit 1052.

When the image determination unit 1052 generates a post-performance image, an image (character) is placed on a virtual spiral along an arrow as shown in FIG. 8 in order from "beginning of performance" to "end of performance". Be placed.

For example, the "start of performance" position and the "end of performance" position are fixed, and the length of the spiral orbital pattern (second condition) increases as the performance time increases. ..

The image information output unit 1053 arranges an image (character) associated with the performance information on the trajectory pattern (second condition) generated by the image determination unit 1052 to generate a real-time image or a post-performance image, and generates an image. Generates image information, which is information for display.

The condition determination unit 1054 determines whether or not the performance data acquired via the communication unit 103 satisfies the set conditions. When the condition determination unit 1054 determines that the performance data acquired via the communication unit 103 satisfies the set condition, the condition determination unit 1054 notifies the image determination unit 1052 to that effect. This condition is set (stored) in, for example, RAM 1061.

When the image determination unit 1052 receives a notification from the condition determination unit 1054 that the condition is satisfied, the image determination unit 1052 switches the image group (character group) from the first image group to the second image group. Specifically, the image determination unit 1052 is related to the performance information following the performance information determined to satisfy the condition, and is different from the image group (for example, the second image) based on the information determined by the performance determination unit 1051. Determine any image data (character) contained in the group).

The performance end determination unit 1055 determines whether or not the performance has ended by at least not accepting the input of the performance information for a certain period of time or receiving the information indicating the end of the performance via the communication unit 103. do.

The condition registration unit 1056 is used by the condition determination unit 1054, and the conditions stored in the RAM 1061 can be arbitrarily changed.

Each unit (operation unit 101, display unit 102, communication unit 103, sound output unit 104, control unit 105, and storage unit 106) described above is communicably connected by the bus 107 and is required between the units. It is possible to exchange various data.

<1-1-4> Speaker In the speaker 3, the sounding unit 40 is a portion that outputs sound. The speaker 3 receives a sound signal from the HP jack 44 by connecting to the HP jack 44 of the electronic musical instrument 2, and outputs the sound.

<1-1-5> Display device The display device 4 has a liquid crystal monitor as a display unit 401 and can display an image. The display device 4 is wiredly connected to the communication unit 103 of the information processing device 1 and receives data from the information processing device 1. Then, the display device 4 displays an image based on the data on the display unit 401.

<1-2> Operation <1-2-1> Flow chart Next, the operation of the information processing apparatus 1 will be described with reference to FIG. 9. FIG. 9 is a flowchart showing the operation of the information processing apparatus 1.

In the following description, it is assumed that the communication unit 103 of the information processing apparatus 1 and the communication unit 80 of the electronic musical instrument 2 are connected. Further, as will be described below, it is assumed that the control unit 105 of the information processing device 1 has started an application for displaying an image on the display device 4.

When the user presses the key 11 of the electronic musical instrument 2 for playing, the key pressing detection unit 50 detects the key pressing of the key 11.

Then, when the key 11 is detected, the control unit 70 of the electronic musical instrument 2 obtains performance information (for example, pitch information (note number, velocity value, etc.)) corresponding to the key 11. Generate note-on) including. Then, the control unit 70 of the electronic musical instrument 2 controls the speaker 3 (or the sounding unit 40) to produce a sound according to the performance information.

On the other hand, when the user releases the key 11 that has been pressed, the key release detection unit 50 detects the release of the key 11.

Then, when the release of the key 11 is detected, the control unit 70 of the electronic musical instrument 2 generates performance information (for example, note-off) corresponding to the released key 11. Then, the control unit 70 of the electronic musical instrument 2 controls the speaker 3 (or the sounding unit 40) to mute the sound according to the performance information.

Then, the performance information generated by the control unit 70 of the electronic musical instrument 2 is transmitted from the communication unit 80 of the electronic musical instrument 2 to the communication unit 103 of the information processing device 1.

≪S1001≫
As shown in FIG. 9, the control unit 105 of the information processing device 1 determines whether or not the performance information transmitted from the electronic musical instrument 2 is received by the communication unit 103 of the information processing device 1 and the performance information is input (). Step S1001). The control unit 105 of the information processing apparatus 1 proceeds to step S1006 when there is no input of performance information (step S1001: NO).

≪S1002≫
Then, when the performance information transmitted from the electronic musical instrument 2 is received by the communication unit 103 of the information processing device 1 (step S1001: YES), the control unit 105 of the information processing device 1 accepts the input of the performance information. Is executed, and the performance determination process (may be described as music analysis or music analysis process) of step S1002 is executed.

Specifically, the control unit 105 (performance determination unit 1051) of the information processing apparatus 1 is based on the received performance information as the performance determination process in step S1002, for example, the tonality of the song, the sound name, the chord type, and the velocity. It determines the value, note length, syncopation, chord function, and chord progression.

However, as the performance determination process in step S1002, the control unit 105 (performance determination unit 1051) of the information processing apparatus 1 uses the tonality, sound name, chord type, velocity value, sound length, syncopation, chord function, and chord of the song. Any one of the determination of progress may be performed.

The details of what is specifically determined as the performance determination process in step S1002 will be described in the next step S1003.

≪S1003≫
When the control unit 105 (image determination unit 1052) of the information processing apparatus 1 finishes the performance determination process in step S1002, the process proceeds to step S1003 and generates a real-time image according to the performance information in accordance with the timing at which the performance information is received. do. Then, the control unit 105 (image information output unit 1053) of the information processing device 1 executes a real-time image output process of outputting and displaying the image to the display device 4.

It should be noted that the above-mentioned "according to the timing at which the performance information is received" does not mean simultaneous time, but means each time the performance information is received.

In the present embodiment, the basic image stored in the storage unit 106 of the information processing device 1 is processed by computer graphics (CG) to create a real-time image and displayed on the display device 4. The real-time image to be displayed on the display device 4 does not have to be limited to such a method, and may be a real-time image according to the performance information.

≪S1004≫
The control unit 105 (condition determination unit 1054) of the information processing device 1 determines whether or not the performance data satisfies the set conditions. The set conditions are stored in, for example, RAM 1061. As an example of determining whether or not the set condition is satisfied, the user registers (stores) a specific phrase in the RAM 1061 or the like, and the control unit 105 of the information processing apparatus 1 is registered in the RAM 1061 based on the performance information. It may be determined whether or not the registered phrase has been played. A "phrase" is a unit consisting of several musical notes. Further, the "phrase" may refer to a set of a plurality of notes (or note numbers). The note is a MIDI term, and represents the number (or pitch) of the key 11 of the electronic musical instrument 2. Further, as an example of determining whether or not the set condition is satisfied, it may be determined whether or not the song being played has been transposed. In addition, as an example of determining whether or not this set condition is satisfied, the rhythm of the song being played (rapid change in BPM, sudden change in the average of the latest velocity value, change in time signature, detection of break). May be determined whether or not has changed. Further, as an example of determining whether or not the set condition is satisfied, it may be determined whether or not the user has pressed a specific pedal (not shown) attached to the electronic musical instrument 2. Further, as an example of determining whether or not the set condition is satisfied, it may be determined whether or not the phrase being played is raised by a semitone as compared with the immediately preceding phrase.

≪S1005≫
When the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing apparatus 1 determines that the performance data satisfies the set conditions (S1004: YES), the drawing setting of the real-time image is changed. do. Specifically, the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing apparatus 1 is based on the performance data acquired after the acquisition of the performance data determined to satisfy the set conditions. When generating a real-time image according to the information that can be determined, the image group is switched and any image data included in the different image groups is displayed.

≪S1006≫
When the performance information transmitted from the electronic musical instrument 2 is not received by the communication unit 103 of the information processing device 1, the control unit 105 (performance end determination unit 1055) of the information processing device 1 receives the performance data (step S1001: NO). When it is determined that the set conditions are not satisfied (S1004: NO), or after step S1005, the performance end determination process is performed. Specifically, the control unit 105 of the information processing apparatus 1 determines whether or not a predetermined time elapses without inputting performance information. If the predetermined time has not elapsed without inputting the performance information (step S1006: NO), the control unit 105 of the information processing apparatus 1 proceeds to step S1001 again and determines whether or not the performance information has been input. do.

≪S1007≫
The control unit 105 (performance end determination unit 1055) of the information processing apparatus 1 has elapsed a predetermined time without inputting performance information (step S1006: YES), that is, when reception of performance information is not detected for a predetermined time. It is determined that the performance has ended. When the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing device 1 determines that the performance has been completed, the process proceeds to step S1007 and outputs the post-performance image to the display device 4. Post-performance image output processing to be displayed is performed, and the entire processing is completed.

Further, when the present embodiment is applied when the accompaniment data is automatically played and the performer is made to specify the melody, it is determined that the performance is finished when the automatic playback of the accompaniment data is finished, and the post-performance image. Output processing may be performed. That is, the reason why the control unit 105 of the information processing device 1 determines the end of the performance by the performer may be any reason, and the control unit 105 of the information processing device 1 determines the end of the performance by the performer. At that point, post-performance image output processing may be executed.

On the other hand, if the predetermined time has not elapsed without inputting the performance information (step S1006: NO), the control unit 105 of the information processing apparatus 1 proceeds to step S1001 again, and whether or not the performance information has been input. To judge.

<1-2-2> Specific Example Next, a specific example of the operation of the information processing apparatus 1 will be described.

In a specific example, for example, when the user plays "For Elise", a case where the image group is switched by an arbitrary phrase of the song will be described.

<1-2-2-1> Phrase A phrase in which a user switches an image group will be described with reference to FIG. FIG. 10 shows a musical score played by the user.

For example, when the user plays "for Elise" and the image group is switched by the third phrase shown in FIG. 10, the phrase immediately before the third phrase is registered in the information processing apparatus 1 as the second phrase. .. In addition, "registration" has the same meaning as "memory". In addition, the phrase before the second phrase is labeled as the first phrase.

<1-2-2-2> Registration method Next, a phrase registration method will be described with reference to FIG. FIG. 11 is a flowchart showing an example of a method of registering “set conditions” for determination by the control unit 105 of the information processing apparatus 1 in step S1004 of FIG. In order for the user to switch the image group with the third phrase shown in FIG. 10, it is necessary to register the second phrase in the information processing apparatus 1.

≪S2001≫
The user selects, for example, the function of registering a drawing setting change phrase by pressing the registration button (or registration icon) from the menu screen of the application displayed on the display unit 102 of the information processing apparatus 1. The function of registering the drawing setting change phrase is executed by, for example, the condition registration unit 1056.

≪S2002≫
When the function of registering the drawing setting change phrase is selected, the control unit 105 of the information processing device 1 receives the performance information transmitted from the electronic musical instrument 2 by the communication unit 103 of the information processing device 1, and inputs the performance information. It is determined whether or not there is (step S2002). The control unit 105 of the information processing apparatus 1 proceeds to step S2004 when there is no input of performance information (step S2002: NO).

≪S2003≫
When the performance information is input (step S2002: YES), the control unit 105 of the information processing device 1 stores the performance information in the storage unit 106 (for example, RAM 1061) of the information processing device 1. The performance information stored in the RAM 1061 is, for example, a "note (or note number)".

≪S2004≫
The control unit 105 of the information processing apparatus 1 determines whether or not "a predetermined time has elapsed without performance information" or "the maximum number of notes has been reached". The maximum number of notes is the maximum number of notes that can be stored as a registered phrase, and can be changed as appropriate. The registration phrase is a general term for "phrases" registered by the user.

When the control unit 105 of the information processing apparatus 1 determines that "a predetermined time has not elapsed without performance information" or "the maximum number of notes has not been reached" (S2004: NO), step S2002. repeat.

≪S2005≫
When the control unit 105 of the information processing apparatus 1 determines that "a predetermined time has elapsed without performance information" or "the maximum number of notes has been reached" (S2004: YES), the RAM 1061 is contacted. A plurality of stored notes are registered (stored) in RAM 1061 as "phrases".

Then, for example, when the user selects an image group, the control unit 105 of the information processing apparatus 1 associates the "phrase" registered in the RAM 1061 with the image setting file. In other words, the control unit 105 of the information processing apparatus 1 designates an image group when the "phrase" is detected. The association information between the "phrase" and the image group is stored in, for example, RAM 1061.

<1-2-2-3> Phrase detection method Next, a phrase detection method will be described with reference to FIG. FIG. 12 is a flowchart showing a specific example of the determination method of the “set condition” in step S1004 of FIG. The phrase is detected in step S1004 shown in FIG.

In step S1004 shown in FIG. 9, the control unit 105 of the information processing apparatus 1 determines whether or not the performance data satisfies the set condition. In this specific example, when determining whether or not the set condition is satisfied, it is determined whether or not the set phrase has been played from the performance data. Hereinafter, a specific flowchart for determining whether or not the phrase set from the performance data has been played will be described.

≪S3001≫
The control unit 105 of the information processing apparatus 1 determines whether or not the note included in the input performance information and the note of the first note of the registered phrase stored in the RAM 1061 match.

≪S3002≫
When the control unit 105 of the information processing apparatus 1 determines that the note included in the input performance information and the note of the first note of the registered phrase stored in the RAM 1061 match (S3001: YES). ), It is determined whether or not the note of the registered phrase stored in the RAM 1061 is completed.
≪S3003≫
When the control unit 105 of the information processing apparatus 1 determines that the note of the registered phrase stored in the RAM 1061 has been completed (S3002: YES), the control unit 105 determines that the input performance information is the registered phrase. That is, the control unit 105 of the information processing apparatus 1 detects the "phrase".

As a result, in step S1004 shown in FIG. 9, the control unit 105 of the information processing apparatus 1 determines that the performance data satisfies the set condition (S1004: YES).

≪S3004≫
When the control unit 105 of the information processing apparatus 1 determines that the note of the registered phrase stored in the RAM 1061 has not been completed (S3002: NO), the input performance information (note) and the note of the registered phrase Calculate the error (difference in note numbers). The control unit 105 of the information processing apparatus 1 accumulates errors each time step S3004 is performed. Such a cumulative total is described as, for example, a cumulative error.

≪S3005≫
The control unit 105 of the information processing apparatus 1 determines whether or not the cumulative error calculated in step S3004 is larger than the threshold value (cumulative error> threshold value). This threshold value is stored in, for example, RAM 1061 or ROM 1062 and can be set as appropriate.

For example, the phrase played by the user may not be a registered phrase but may happen to contain the same sound as the registered phrase, or the user may be playing the registered phrase but making a mistake. If the phrase played by the user is not a registered phrase, switching image groups is not preferable because it is not the behavior the user wants. Therefore, the control unit 105 of the information processing apparatus 1 needs to appropriately determine in step S3005 that the phrase played by the user is not a registered phrase. Further, it is terrible for the user to determine that the phrase is not a registered phrase just by making a mistake even though the user is playing the registered phrase. One of the purposes of the performance data display system is to support the practice of the user's musical instrument in the first place. Therefore, it is necessary to assume that the user cannot play the registered phrase properly. Therefore, the control unit 105 of the information processing apparatus 1 needs to appropriately determine in step S3005 that the user has tried to play the registered phrase.

≪S3006≫
When the control unit 105 of the information processing apparatus 1 does not determine that the cumulative error is larger than the threshold value (S3005: NO), the control unit 105 selects the note next to the registered phrase. Then, the process proceeds to step S3002.

≪S3007≫
When the control unit 105 of the information processing apparatus 1 determines that the cumulative error is larger than the threshold value (S3005: YES), the control unit 105 determines that the input performance information is not a registered phrase.

As a result, in step S1004 shown in FIG. 9, the control unit 105 of the information processing apparatus 1 determines that the performance data does not satisfy the set condition (S1004: NO). That is, the control unit 105 of the information processing apparatus 1 could not detect the "phrase".

<1-2-2-4> Example of drawing a real-time image Subsequently, using FIGS. 13 and 14, the user plays the second phrase in FIG. 10, and then the third phrase to switch the image group. I will explain the situation. FIG. 13 shows a real-time image when the user is playing the second phrase in FIG. FIG. 14 shows a real-time image when the user is playing the third phrase in FIG. Here, it is assumed that the second phrase shown in FIG. 10 is registered in the RAM 1061 by the operation shown in FIG. Then, the control unit 105 of the information processing apparatus 1 performs the operations shown in FIGS. 9 and 12. Further, here, as an example, an example in which the first image group is switched to the second image group by playing the registration phrase will be described.

As shown in FIG. 13, when the user plays the second phrase in FIG. 10, the control unit 105 of the information processing apparatus 1 displays an image (insect character) included in the first image group as a real-time image in the display apparatus 4. (See B1 to B10 in the figure). At this time, for example, an image (insect character) is displayed along the arrow A1. The control unit 105 of the information processing apparatus 1 performs the operation described with reference to FIG. 12 while displaying a real-time image, and determines that the second phrase is a registered phrase.

When the user plays the second phrase, the control unit 105 of the information processing apparatus 1 switches the image group from the first image group to the second image group. As shown in FIG. 14, when the user plays the third phrase in FIG. 10, the control unit 105 of the information processing apparatus 1 displays an image (animal character) included in the second image group as a real-time image in the display apparatus 4. (See C1 to C11 in the figure). At this time, for example, an image (animal character) is displayed along the arrow A2.

In this way, the user registers the phrase for switching the image group in the information processing apparatus 1 in advance, so that the information processing apparatus 1 recognizes the registered phrase and switches the image group. Therefore, the user can display an arbitrary image (character) at an arbitrary timing.

<1-2-2-5> Example of Drawing a Post-Performance Image Next, with reference to FIG. 15, a post-performance image after the user finishes playing the music shown in FIG. 10 will be described. FIG. 15 shows a post-performance image after the user has finished playing the song shown in FIG. Here, as in the example described with reference to FIGS. 13 and 14, an example in which the first image group is switched to the second image group by playing the registration phrase will be described as an example.

When the control unit 105 of the information processing device 1 determines that the user has finished playing, the control unit 105 causes the display device 4 to display the image after the performance. As shown in FIG. 15, the control unit 105 of the information processing apparatus 1 arranges the images (insect characters) of the first group at the locations corresponding to the second phrases (see B1 to B10 in the figure), and the first The second group of images (animal characters) are placed at the locations corresponding to the three phrases (see C1 to C11 in the figure).

Thereby, the same effect as the above-mentioned effect can be obtained.

<1-3> Effect According to the above-described embodiment, in the performance data display system that draws an image according to the user's performance (music), when it is determined that the performance data satisfies the set condition, the image to be drawn is drawn. Switch the image group used for.

For example, a performance data display system that draws an image according to a user's performance (music) can arouse the user's willingness to practice an electronic musical instrument. Therefore, by further adopting the above-described embodiment, it is possible to change the drawing setting without performing an operation unrelated to the performance for drawing. Therefore, it is possible to widen the range of drawing expression without performing a peculiar operation unrelated to the performance for drawing, and it is possible to surely produce an expression change. As a result, the expressive power of the image drawn by the performance can be improved while concentrating the user on the performance. As a result, the user's willingness to practice the electronic musical instrument can be further stimulated.

<2> Others In the above-described embodiment, the control unit 105 of the information processing apparatus 1 switches the image group when the drawing setting is changed in step S1005 of FIG. However, the present invention is not limited to this, for example, "change of brightness", "change of image (character) size", "background color (real-time image or background color of post-play image), image (character)". It may be "change of color table", "change of type of image (bonus character) displayed when a specific condition is satisfied", "change of movement of image (character)", or the like. The background color (the background color of the real-time image or the image after the performance) and the color table of the image (character) are stored in, for example, ROM 1062. Further, the image (bonus character) displayed when a specific condition is satisfied is, for example, a real-time image when the control unit 105 of the information processing apparatus 1 determines that the rhythm of the user's performance is stable. This is the image to be displayed. Further, the image (character) may be moved by animation or the like. Further, the same image (character) may be included in a plurality of image groups.

Further, in the above-described embodiment, the control unit 105 of the information processing apparatus 1 may determine in step S1004 of FIG. 9 whether or not the music being played has been transposed. In this case, since the accuracy of the key determination is low immediately after the start of the performance, it is not necessary to change the drawing setting by detecting the modulation during a certain period of time from the start of the performance.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

In addition, the above-mentioned embodiment includes the following additional notes.

≪Appendix 1≫
To the computer of the information processing device
The first image data is displayed according to the information that can be determined based on the acquired first performance data.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set condition.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the information that can be determined based on the first performance data. Even if they are the same, the second image data having a different aspect from the first image data is displayed.
program.

≪Appendix 2≫
The first image data is included in the first image group, and is included in the first image group.
The second image data is included in the second image group, and is included in the second image group.
The same image data may be included in the first image group and the second image group.
The program described in Appendix 1.

≪Appendix 3≫
When it is determined that the condition for satisfying the set condition is satisfied, at least when it is determined that the phrase set from the acquired second performance data has been played, or when it is determined that the key has been transposed. And, if it is determined that the rhythm has changed, including any of
The program described in Appendix 1 or 2.

≪Appendix 4≫
The information that can be determined includes at least one of a note name and a chord type.
The program according to any one of Supplementary note 1 to 3.

≪Appendix 5≫
Lets you judge whether the performance is over and
If it is not determined that the performance is over, the image data is displayed according to the set first condition.
The program according to any one of Supplementary note 1 to 4, which displays image data according to a set second condition when it is determined that the performance has ended.

≪Appendix 6≫
To the computer of the information processing device
The first image data is displayed according to the information that can be determined based on the acquired first performance data.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set condition.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data, even if it is the same as the information that can be determined based on the first performance data. Even if there is, the second image data in a mode different from the first image data is displayed.
Method.

≪Appendix 7≫
Display device and
The processor, including the processor,
According to the information that can be determined based on the acquired first performance data, the display of the first image data on the display device is instructed.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set conditions.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the information that can be determined based on the first performance data. Even if they are the same, the display of the second image data in a mode different from that of the first image data on the display device is instructed.
Electronics.

≪Appendix 8≫
Equipped with an electronic musical instrument and a display device,
The electronic musical instrument is
The first performance data is generated in response to the first performance operation by the user, and the generated first performance data is output to the display device.
The second performance data is generated according to the second performance operation by the user, and the generated second performance data is output to the display device.
The third performance data is generated according to the third performance operation by the user, and the generated third performance data is output to the display device.
The display device is
The first image data is displayed according to the information that can be determined based on the acquired first performance data.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set condition.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the information that can be determined based on the first performance data. Even if it is the same as, the second image data in an aspect different from the first image data is displayed.
Performance data display system.

1 ... Information processing device 2 ... Electronic musical instrument 3 ... Speaker 4 ... Display device 11 ... Key 20 ... Display unit 30 ... Operation unit 40 ... Sound unit 41 ... SP amplifier 42 ... Speaker 43 ... HP amplifier 44 ... HP jack 45 ... HP jack Insertion detection unit 50 ... Key press detection unit 60 ... Storage unit 61 ... ROM
62 ... RAM
70 ... Control unit 80 ... Communication unit 100 ... Bus 101 ... Operation unit 102 ... Display unit 103 ... Communication unit 104 ... Sound output unit 105 ... Control unit 106 ... Storage unit 107 ... Bus 401 ... Display unit 1051 ... Performance determination unit 1052 ... Image determination unit 1053 ... Image information output unit 1054 ... Condition determination unit 1055 ... Performance end determination unit 1056 ... Condition registration unit 1061 ... RAM
1062 ... ROM
10621 ... Program 10622 ... Table

Claims (8)

To the computer of the information processing device
The first image data is displayed according to the information that can be determined based on the acquired first performance data.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set condition.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the information that can be determined based on the first performance data. Even if they are the same, the second image data having a different aspect from the first image data is displayed.
program. The first image data is included in the first image group, and is included in the first image group.
The second image data is included in the second image group that is changed from the first image group when it is determined that the set condition is satisfied .
The program according to claim 1. When it is determined that the condition for satisfying the set condition is satisfied, at least when it is determined that the phrase set from the acquired second performance data has been played, or when it is determined that the key has been transposed. And, if it is determined that the rhythm has changed, including any of
The program according to claim 1 or 2. The information that can be determined includes at least one of a note name and a chord type.
The program according to any one of claims 1 to 3. Lets you judge whether the performance is over and
If it is not determined that the performance is over, the image data is displayed according to the set first condition.
The program according to any one of claims 1 to 4, wherein the image data is displayed according to the set second condition when it is determined that the performance has ended. To the computer of the information processing device
The first image data is displayed according to the information that can be determined based on the acquired first performance data.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set condition.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the information that can be determined based on the first performance data. Even if they are the same , the second image data having a different aspect from the first image data is displayed.
Method. Display device and
The processor, including the processor,
According to the information that can be determined based on the acquired first performance data, the display of the first image data on the display device is instructed.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set conditions.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the information that can be determined based on the first performance data. Even if they are the same, the display of the second image data in a mode different from that of the first image data on the display device is instructed.
Electronics. Equipped with an electronic musical instrument and a display device,
The electronic musical instrument is
The first performance data is generated in response to the first performance operation by the user, and the generated first performance data is output to the display device.
The second performance data is generated according to the second performance operation by the user, and the generated second performance data is output to the display device.
The third performance data is generated according to the third performance operation by the user, and the generated third performance data is output to the display device.
The display device is
The first image data is displayed according to the information that can be determined based on the acquired first performance data.
It is determined whether or not the second performance data acquired after the acquisition of the first performance data satisfies the set condition.
When it is determined that the set condition is satisfied, the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the information that can be determined based on the first performance data. Even if it is the same as, the second image data in an aspect different from the first image data is displayed.
Performance data display system.

Images