CN113093975A

CN113093975A - Display method, electronic device, performance data display system, and storage medium

Info

Publication number: CN113093975A
Application number: CN202011490569.2A
Authority: CN
Inventors: 加福滋; 奥田广子
Original assignee: Casio Computer Co Ltd
Current assignee: Casio Computer Co Ltd
Priority date: 2019-12-23
Filing date: 2020-12-17
Publication date: 2021-07-09
Also published as: JP7070538B2; JP2021099460A

Abstract

An object of the present invention is to provide a display method, an electronic device, a performance data display system, and a storage medium that are excellent in the visual expression of performance. A display method, characterized in that a processor of an information processing device performs processing of displaying first image data corresponding to information that can be determined based on acquired first performance data; Whether or not the second performance data acquired after the acquisition satisfies the set conditions; if it is determined that the set conditions are satisfied, even information that can be determined based on the third performance data acquired after the acquisition of the second performance data Similar to the information that can be determined from the above-mentioned first performance data, second image data in a different form from the above-mentioned first image data is also displayed.

Description

Display method, electronic device, performance data display system, and storage medium

Technical Field

The invention relates to a display method, an electronic apparatus, a performance data display system, and a storage medium.

Background

For example, there is known a computer graphics moving image generation system that moves an image object such as a dancer (dancer) in synchronization with the performance of music such as MIDI (Musical Instrument Digital Interface) music and can generate a moving image that changes integrally with the music as a result of the progress of Musical tones in addition to the idea of the music (for example, japanese patent application laid-open No. 11-224084).

In the computer graphics moving image generation system disclosed in japanese patent application laid-open No. 11-224084, since a character (character) representing a dancer is simply dynamically displayed during playing, there is a problem of lack of expressiveness.

Disclosure of Invention

According to the display method of the embodiment, the following processing is performed by a processor of an information processing apparatus: displaying the 1st image data in accordance with information that can be determined from the acquired 1st performance data; determining whether or not the 2 nd performance data acquired after the acquisition of the 1st performance data satisfies a set condition; when it is determined that the set condition is satisfied, even if the information that can be determined based on the 3 rd performance data acquired after the 2 nd performance data is acquired is the same as the information that can be determined based on the 1st performance data, the 2 nd image data having a different form from the 1st image data is displayed.

Drawings

Fig. 1 is a diagram schematically illustrating an example of a performance data display system of an application example.

Fig. 2 is a diagram showing an outline of the performance data display system according to the embodiment.

Fig. 3 is a diagram showing hardware of a control system of an electronic musical instrument according to the embodiment.

Fig. 4 is a diagram showing hardware of an information processing apparatus according to an embodiment.

Fig. 5 is a diagram showing an example of a table stored in the ROM table.

Fig. 6 is a diagram showing an example of display of an image (character).

Fig. 7 is a diagram showing an example of the trajectory pattern of an image (character) in a live view displayed on the display unit of the display device.

Fig. 8 is a diagram showing a spiral line as an example of the track pattern generated by the image determination unit.

Fig. 9 is a flowchart showing an operation of the information processing apparatus.

Fig. 10 shows a spectral plane played by a user.

Fig. 11 is a flowchart showing an example of the registration method of the "set condition" for the control unit to determine in step S1004 of fig. 9.

Fig. 12 is a flowchart showing a specific example of the determination method of the "set condition" in step S1004 of fig. 9.

Fig. 13 shows a real-time image in the case where the user plays the 2 nd phrase in fig. 10.

Fig. 14 shows a real-time image in the case where the user plays phrase 3 in fig. 10.

Fig. 15 shows a post-performance image after the user finishes the performance of the tune shown in fig. 10.

Detailed Description

Hereinafter, an embodiment of one embodiment of the present invention will be described with reference to the drawings. However, the embodiments described below are merely illustrative in all aspects of the present invention. It is needless to say that various improvements and modifications can be made without departing from the scope of the present invention. That is, in the implementation of the present invention, the specific configuration corresponding to the embodiment can be appropriately adopted. The data appearing in the embodiment is described in natural language, but more specifically, it is specified in a simulation language, commands, parameters, machine language, and the like that can be recognized by a computer.

<0> application example

First, an example of a scenario to which the present invention is applied will be described with reference to fig. 1. Fig. 1 schematically illustrates an example of a performance data display system of an application example. The performance data display system envisions the following cases: an image (drawing) is drawn in real time in accordance with a performance (music) of a user (performer), thereby stimulating a user's exercise intention on the electronic musical instrument.

< 0-1 > overview of application example

Before explaining the configuration of the performance data display system, the outline of the performance data display system will be explained. For example, the performance data display system analyzes performance data received from an electronic musical instrument or the like capable of outputting a user's performance as performance data (for example, MIDI data), and converts the user's performance into an image. For example, note number information included in the musical performance data may be used to determine note name information and chord type information.

< 0-2 > Structure of application example

The structure of the application example will be described with reference to fig. 1. Fig. 1 is a diagram showing an example of the configuration of a performance data display system according to an application example.

As shown in fig. 1, the performance data display system includes an electronic musical instrument, an information processing apparatus, and a display apparatus.

The electronic musical instrument converts the user's performance into performance data (e.g., MIDI data). Then, the electronic musical instrument outputs the performance data to the information processing device.

More specifically, the electronic musical instrument generates the 1st performance data according to the 1st performance operation by the user, and outputs the generated 1st performance data. Further, the electronic musical instrument generates the 2 nd performance data according to the 2 nd performance operation of the user, and outputs the generated 2 nd performance data. For example, the 2 nd performance operation is an operation following the 1st performance operation. The electronic musical instrument generates 3 rd performance data according to the 3 rd performance operation by the user, and outputs the generated 3 rd performance data. For example, the 3 rd performance operation is an operation following the 2 nd performance operation. The electronic musical instrument is, for example, a keyboard, an electronic piano, an electric guitar, an electronic violin, or the like. However, the electronic musical instrument is not limited to the above, and may be appropriately selected according to the embodiment.

The information processing device analyzes the received performance data and converts the performance data into image data. More specifically, the information processing apparatus outputs the 1st image (character) data based on information (for example, a sound name, a chord type, and the like) that can be determined based on the acquired 1st performance data. The information processing device determines whether or not the 2 nd performance data acquired after the acquisition of the 1st performance data satisfies a set condition, and outputs the 2 nd image (character) data in a form different from that of the 1st image data even if information (for example, a sound name, a chord type, and the like) that can be determined based on the 3 rd performance data acquired after the acquisition of the 2 nd performance data is the same as information that can be determined based on the 1st performance data when it is determined that the set condition is satisfied. The information processing apparatus is, for example, a PC (personal computer). The term "set" or "registered" basically has the same meaning as "stored". Specific examples of the "set conditions" will be described later.

The display device displays an image based on a performance of a user. More specifically, the display device displays image data received from the information processing device on the display unit.

< 0-3 > actions of application example

Next, an example of the operation of the performance data display system of the application example will be described.

When the user starts playing with the electronic musical instrument, the information processing apparatus performs music analysis on the playing. Specifically, the information processing apparatus determines an image (for example, a monarch character) included in the 1st image group (for example, a character group including a plurality of insect characters) based on the musical name (or chord) included in the musical performance data, and draws the image in real time along a trajectory in accordance with the set 1st condition (for example, display corresponding to chord progression). Thereby, the images (characters) included in the 1st image group are displayed on the display device in real time. In addition, regarding "real time" here, the timing of performance and the timing of display of an image (character) do not necessarily need to be the same timing. Further, an image that displays an image (character) in accordance with the performance may be written as a real-time image, for example. That is, a plurality of images (characters) are drawn in the live image. The method of rendering a real-time image will be described later.

When a user performs a musical performance satisfying a condition set (stored or registered) in the information processing device while playing the electronic musical instrument, the information processing device switches from the 1st image group (for example, a character group including a plurality of insect characters) to the 2 nd image group (for example, a character group including a plurality of animal characters), determines an image (for example, a character of giraffe) included in the 2 nd image group (for example, a character group including a plurality of animal characters) based on a note name (or a chord) included in the musical performance data, and draws the image in real time along a trajectory according to the set 1st condition (for example, a display corresponding to chord progression). Thereby, the images (characters) included in the 2 nd image group are displayed on the display device in real time. The switching of the image group is also referred to as "drawing setting change".

Further, after the user finishes the performance of the electronic musical instrument, the information processing apparatus causes the display apparatus to display the image data in accordance with the set 2 nd condition (e.g., the spiral display). Such an image may be referred to as a post-performance image, for example. The post-performance image is an image obtained by drawing an image (character) analyzed from the start to the end of the performance of the user under the condition 2 (for example, spiral display).

< 0-4 > Effect of application example

As described above, according to the performance data display system of the application example, the set of images to be displayed can be changed by satisfying the predetermined condition while the user is playing the electronic musical instrument. In other words, according to the performance data display system of the application example, it is possible to appropriately perform setting (drawing setting) of an image without performing a unique operation that is not related to a performance during the performance of a user. Therefore, the performance data display system can generate an image with high expression force in a state where the user is concentrating on the performance.

<1> embodiment mode

< 1-1 > Structure

< 1-1-1 > Performance data display System

Hereinafter, a performance data display system according to an embodiment will be described with reference to fig. 2.

As shown in fig. 2, the performance data display system includes, for example, an information processing device (electronic apparatus) 1, an electronic musical instrument 2, a speaker 3, and a display device 4.

< 1-1-2 > electronic musical instrument

Hereinafter, an electronic musical instrument according to an embodiment will be described with reference to fig. 2 and 3. Fig. 3 is a diagram showing hardware of a control system of an electronic musical instrument according to the embodiment.

As shown in fig. 2 and 3, the electronic musical instrument 2 to which the information processing device 1 of the present embodiment is connected is an electronic keyboard musical instrument such as an electronic piano, an electronic synthesizer, or an electronic organ.

The electronic musical instrument 2 includes a plurality of keys 11, a display unit 20, an operation unit 30, and a sound emitting unit 40.

As shown in fig. 3, the electronic musical instrument 2 includes a key detection unit 50, a storage unit 60, a control unit 70, and a communication unit 80.

The keys 11 are operation members for designating the pitch of the sound by the player, and the player presses and releases the keys 11, so that the electronic musical instrument 2 sounds and silences the sound corresponding to the designated pitch.

The display unit 20 includes, for example, a liquid crystal monitor with a touch panel, and is a portion for displaying a message or the like in accordance with the operation of the operation unit 30 by the player.

In the present embodiment, since the display unit 20 has a touch panel function, the display unit 20 can serve as a part of the operation unit 30.

The operation unit 30 has operation buttons for the performer to perform various settings and the like, and is a part for performing various setting operations such as volume adjustment and the like.

The sound emitting unit 40 is a part that outputs sound, and includes an SP microphone 41 (speaker microphone), a speaker 42, an HP microphone 43 (headphone microphone), an HP jack 44 (headphone jack) into which a plug of the speaker 3 (or headphone) is inserted, and an HP jack insertion detecting unit 45 that detects that the plug of the speaker 3 (or headphone) is inserted into the HP jack 44.

When the plug of the speaker 3 (or the earphone) is inserted into the HP jack 44, the HP jack insertion detector 45 detects the insertion, and outputs sound on the HP jack side, whereas when the HP jack insertion detector 45 does not detect the insertion of the plug, sound is output on the speaker 42 side.

Note that the sound emitting unit 40 may be used instead of the speaker 3.

The key detection unit 50 is a part for detecting the key depression and key release of the key 11, and for example, a rubber switch is provided below each key 11, and the key depression and key release are detected by the rubber switch.

The storage unit 60 includes a ROM61 as a storage unit dedicated to reading and a RAM62 as a storage unit readable and writable.

The ROM61 stores, for example, a control program (for example, a program for a learning mode (lesson mode) program) executed by the control unit 70, various data tables, and the like.

The RAM62 is a storage area for storing data such as musical tone data and musical composition data corresponding to each key 11, and also functions as a temporary storage area for developing data and control programs generated by the control unit 70 during performance.

The control unit 70 is a part responsible for controlling the entire electronic musical instrument 2. The control unit 70 is, for example, an lsi (large Scale integration).

The control unit 70 performs, for example, control for sounding a musical sound from the sound generating unit 40 in response to a key depression of the key 11, control for muting a musical sound sounded from the sound generating unit 40 in response to a key depression of the key 11, and the like.

The communication unit 80 includes a wireless unit or a wired unit for communicating with an external device, and in the present embodiment, the information processing apparatus 1 is connected by a wire, and data is transmitted to the information processing apparatus 1 as described later.

However, the communication unit 80 is not limited to performing only the transmission function, and may further include a reception function.

The above-described components (the display unit 20, the operation unit 30, the sound-emitting unit 40, the key detection unit 50, the storage unit 60, the control unit 70, and the communication unit 80) are communicably connected by a bus 100, and necessary data can be exchanged between the components.

< 1-1-3 > information processing apparatus

Next, the information processing apparatus will be described with reference to fig. 2 and 4. Fig. 4 is a diagram showing hardware of an information processing apparatus according to an embodiment.

In the present embodiment, the information processing device 1 is a PC as shown in fig. 2, and is equipped with an application for displaying an image on the display device 4 as will be described later.

However, the information processing device 1 is not necessarily limited to a PC, and may be a tablet-type portable terminal or the like.

The information processing apparatus 1 may further include a sequencer (sequencer) or the like that receives MIDI chords from the electronic musical instrument 2 and reproduces the music data.

As shown in fig. 4, the information processing device 1 mainly includes an operation unit 101, a display unit 102, a communication unit 103, an audio output unit 104, a control unit 105, and a storage unit 106.

The operation unit 101 includes switches such as a power switch for turning ON/OFF (ON/OFF) the power of the information processing apparatus 1, for example.

The display unit 102 includes a liquid crystal monitor and can display an image. The display unit 102 may have a touch panel function. In this case, the display unit 102 can serve as a part of the operation unit 101. In addition, the display unit 102 may be used instead of the display device 4.

The communication unit 103 includes a wireless unit or a wired unit for performing communication with another device or the like. The communication unit 103 is connected to the communication unit of the electronic musical instrument 2 by wire, for example, and receives data from the electronic musical instrument 2. The communication unit 103 is connected to the communication unit of the display device 4 by wire, for example, and transmits data to the display device 4.

The sound output unit 104 is a part that outputs sound, and includes a speaker, an earphone jack, and the like.

The storage unit 106 includes a ram (random Access memory)1061 and a rom (read Only memory)1062 as readable and writable storage units.

The RAM1061 includes a volatile memory, is a storage area for storing data necessary for operating the control program, and functions as a temporary storage area for expanding data generated by the control unit 105, data (performance information including pitch information) transmitted from the electronic musical instrument 2, and applications.

The ROM1062 is a nonvolatile semiconductor memory such as a flash memory or an eprom (Erasable Programmable Read Only memory) or an EEPROM (registered trademark), and plays a role as a so-called secondary storage device or auxiliary storage device.

The ROM1062 stores a program 10621 and a table 10622. The program 10621 is used for the control unit 105 to perform various processes. The table 10622 is a table in which performance information is associated with images.

The control unit 105 is a part responsible for controlling the entire information processing apparatus 1. The control unit 105 includes, for example, a cpu (central Processing unit). The control unit 105 reads out the program 10621 stored in the ROM1062 of the storage unit 106 and uses the RAM1061 of the storage unit 106 as a work area, thereby comprehensively controlling the information processing apparatus 1.

Next, a functional configuration of the control unit 105 of the information processing device 1 according to the embodiment will be described. The control unit 105 reads and executes the program 10621 stored in the ROM1062, and functions as a performance determination unit 1051, an image determination unit 1052, an image information output unit 1053, a condition determination unit 1054, a performance end determination unit 1055, and a condition registration unit 1056. The performance determination unit 1051, the image determination unit 1052, the image information output unit 1053, the condition determination unit 1054, the performance end determination unit 1055, and the condition registration unit 1056 may be implemented not by a program but by a circuit configuration.

The performance determination unit 1051 determines the tonality (for example, 24 types from the C Major to the B minor), the note name (for example, do, re, mi …), the chord type (for example, Major, Mimor, Sus4, Aug, Dim, 7th, and the like), the dynamics value (numerical value indicating the intensity of a note in the MIDI standard), the length of a note, the score, the chord function, and the progression of a chord, based on the performance information (performance data) received from the electronic musical instrument 2 via the communication unit 103. Note that the method of determining the key tone, the name of the music, the type of chord, the strength value, the length of the sound, the score, the chord function, and the chord progression may be any method.

Each time performance information is received from the electronic musical instrument 2, the image determining unit 1052 determines certain image data (character) included in an image group (character group) based on the information determined by the performance determining unit 1051.

Here, the relationship between the information determined by the performance determination unit 1051 and the image data will be described with reference to fig. 5. Fig. 5 is a diagram showing an example of a table stored in table 10622 of ROM 1062.

As shown in fig. 5, the table 10622 stores, for example, the 1st image group and the 2 nd image group in which images (characters) are set in correspondence with the sound names (here, examples of do, re, mi, fa, so, la, xi). The method of selecting the 1st image group and the 2 nd image group will be described later. Note that the table 10622 is not limited to this, and may also store images corresponding to other names of the sounds (e.g., do # and the like) and the types of the chords. For example, in the 1st image group and the 2 nd image group, images of different forms are assigned even with the same sound name.

For example, when the image determination unit 1052 determines that the sound name included in the received performance information is "do", it determines the image (character) of the "butterfly" in the 1st image group in the table 10622.

Here, a method of displaying an image (character) will be described with reference to fig. 6. Fig. 6 is a diagram showing an example of display of an image (character).

As shown in fig. 6, the image (character) stored in the table 10622 is, for example, a basic form. The image determination unit 1052 determines the size of the image (character) displayed on the display device 4 based on the force value included in the performance information. The dimensions of the basic form are determined as height HB and width WB. Then, for example, the height HC and the width WC are changed by changing the percentages of the height HB and the width WB based on the force value. The image determination unit 1052 may rotate the basic form of the image (character) by R degrees, reverse the left and right, and change the color at random. Although not described in detail here, the image determination unit 1052 may change the size, rotation, inversion, or color as appropriate based on information not shown here.

When displaying the live image, the image determining unit 1052 determines a trajectory pattern (condition 1) in which images (characters) are arranged on the display unit of the display device 4, based on the performance information (for example, chord progression). The track pattern (1 st condition) is stored in the ROM1062, for example. Here, an example of a track pattern when an image (character) is displayed will be described with reference to fig. 7. Fig. 7 is a diagram showing an example of the trajectory pattern of an image (character) in a live view displayed on the display unit 401 of the display device 4.

As an example of the track pattern, as shown in fig. 7, an image (character) is sequentially arranged on a virtual arrow along an arrow a1 in the upper half of the live image (see 1st (u)), and an image (character) is sequentially arranged on a virtual arrow along an arrow a2 in the lower half of the live image (see 1st (l)) following the arrow a 1. Further, each image (character) is arranged so that at least a part of each image (character) is superimposed on the virtual arrows (a1, a2) of the track pattern.

The image determining section 1052 generates a track pattern (condition 2) for displaying the after-performance image on the display device 4 after the performance of the user. The image determining unit 1052 generates a track pattern (condition 2) in which images (characters) corresponding to the performance information are arranged. The image determining unit 1052 determines the length of the track pattern (condition 2) to be played based on the playing time calculated by the playing determining unit 1051. For example, regarding the 1st length, which is the length of the track pattern (2 nd condition) in the case where the performance time is the 1st performance time length, and the 2 nd length, which is the length of the track pattern (2 nd condition) in the case where the performance time is the 2 nd performance time length, in the case where the 1st performance time length is shorter than the 2 nd performance time length, the 1st length is shorter than the 2 nd length.

An example of the track pattern (condition 2) generated by the image determination unit 1052 will be described with reference to fig. 8. Fig. 8 is a diagram showing a spiral line as an example of the track pattern (condition 2) generated by the image determining unit 1052.

When generating the post-performance image, the image determination unit 1052 arranges the images (characters) in order from "the start of the performance" to "the end of the performance" on a virtual spiral line along an arrow shown in fig. 8.

In addition, for example, the position of "start of performance" and the position of "end of performance" are fixed, and as the time of performance becomes longer, the length of the spiral-shaped track pattern (condition 2) is extended.

The image information output unit 1053 arranges an image (character) corresponding to the performance information on the trajectory pattern (condition 2) generated by the image determination unit 1052, generates a live image or a performed image, and generates image information as information for displaying the image.

The condition determination unit 1054 determines whether or not the performance data acquired via the communication unit 103 satisfies a set condition. When determining that the performance data acquired via the communication unit 103 satisfies the set conditions, the condition determination unit 1054 notifies the image determination unit 1052 of the determination. The condition is set (stored) in the RAM1061, for example.

Upon receiving a notification from the condition determination unit 1054 that the condition is satisfied, the image determination unit 1052 switches the image group (the character group) from the 1st image group to the 2 nd image group. Specifically, the image determining unit 1052 determines certain image data (character) included in a different image group (for example, the 2 nd image group) based on information associated with performance information subsequent to the performance information determined to satisfy the condition and determined by the performance determining unit 1051.

The performance end determination unit 1055 determines whether or not a performance has ended by not receiving any of the input of performance information and the reception of information indicating the end of the performance via the communication unit 103 for at least a certain period of time.

The condition registration unit 1056 is used by the condition determination unit 1054, and can arbitrarily change the conditions stored in the RAM 1061.

The above-described units (the operation unit 101, the display unit 102, the communication unit 103, the sound output unit 104, the control unit 105, and the storage unit 106) are communicably connected by a bus 107, and necessary data can be exchanged between the units.

< 1-1-4 > speaker

The speaker 3 is a part to which the sound generating unit 40 outputs sound. The speaker 3 is connected to the HP jack 44 of the electronic musical instrument 2, receives a sound signal from the HP jack 44, and outputs sound.

< 1-1-5 > display device

The display device 4 includes a liquid crystal monitor as the display unit 401, and is capable of displaying an image. The display device 4 is connected to the communication unit 103 of the information processing device 1 by wire, and receives data from the information processing device 1. The display device 4 displays an image based on the data on the display unit 401.

< 1-2 > act

< 1-2-1 > flow chart

Next, the operation of the information processing apparatus 1 will be described with reference to fig. 9. Fig. 9 is a flowchart showing the operation of the information processing apparatus 1.

In the following description, it is assumed that the communication unit 103 of the information processing device 1 and the communication unit 80 of the electronic musical instrument 2 are connected to each other. As described later, the control unit 105 of the information processing apparatus 1 is assumed to start an application for displaying an image on the display device 4.

When the user presses the key 11 of the electronic musical instrument 2 for playing, the key 11 is detected by the key detection unit 50.

When the key 11 is pressed, the control unit 70 of the electronic musical instrument 2 generates performance information (for example, note-on (note on) including pitch information and the like (note number, strength value and the like)) corresponding to the pressed key 11. The control unit 70 of the electronic musical instrument 2 performs control for emitting a sound corresponding to the performance information from the speaker 3 (or the sound emitting unit 40).

On the other hand, when the user releases the pressed key 11, the key release of the key 11 is detected by the key detection unit 50.

When the key release of the key 11 is detected, the control unit 70 of the electronic musical instrument 2 generates performance information (for example, note off) corresponding to the key 11 that is released. The control unit 70 of the electronic musical instrument 2 performs control for muting the sound corresponding to the performance information from the speaker 3 (or the sound emitting unit 40).

The performance information generated by the control unit 70 of the electronic musical instrument 2 is transmitted from the communication unit 80 of the electronic musical instrument 2 to the communication unit 103 of the information processing device 1.

《S1001》

As shown in fig. 9, the control unit 105 of the information processing device 1 determines whether or not the communication unit 103 of the information processing device 1 has received the performance information transmitted from the electronic musical instrument 2 and has input of the performance information (step S1001). When the performance information is not input (no in step S1001), the control unit 105 of the information processing device 1 proceeds to step S1006.

《S1002》

When the communication unit 103 of the information processing device 1 receives the performance information transmitted from the electronic musical instrument 2 (yes in step S1001), the control unit 105 of the information processing device 1 executes a process of receiving the input of the performance information and executes a performance determination process (which may be referred to as music analysis or music analysis process) in step S1002.

Specifically, as the performance determination process in step S1002, the control unit 105 (performance determination unit 1051) of the information processing device 1 determines, for example, the tonality, the sound name, the chord type, the strength value, the length, the score, the chord function, and the chord progression of the tune based on the received performance information.

However, as the performance determination process in step S1002, the control unit 105 (performance determination unit 1051) of the information processing apparatus 1 may perform any one of determination of the key tone, the note name, the chord type, the strength value, the note length, the score, the chord function, and the chord progression of the tune.

The performance determination processing in step S1002 will be described in detail with respect to details such as what is determined, in the following step S1003.

《S1003》

When the performance determination processing in step S1002 is completed, the control unit 105 (the image determination unit 1052) of the information processing device 1 proceeds to step S1003, and generates a real-time image corresponding to the performance information in accordance with the timing at which the performance information is received. The control unit 105 (image information output unit 1053) of the information processing device 1 executes real-time image output processing for outputting and displaying the image on the display device 4.

Note that the above "corresponding to the timing at which the performance information is received" does not mean that the performance information is received at the same time in terms of time, but means that the performance information is received at each time.

In the present embodiment, the real-time image is created by processing the image to be the base stored in the storage unit 106 of the information processing device 1 by Computer Graphics (CG) and displayed on the display device 4, but the method for displaying the real-time image on the display device 4 is not limited to this, and any real-time image corresponding to the performance information may be used.

《S1004》

The control unit 105 (condition determination unit 1054) of the information processing device 1 determines whether or not the performance data satisfies the set conditions. The set conditions are stored in the RAM1061, for example. As an example of determining whether or not the set condition is satisfied, a user may register (store) a specific phrase in the RAM1061 or the like, and the control unit 105 of the information processing apparatus 1 may determine whether or not a performance of the registered phrase registered in the RAM1061 is performed based on the performance information. In addition, a "phrase" is a unit composed of several notes. Further, "phrase" may also refer to a set of a plurality of notes (or phonetic symbol codes). Note is a term of MIDI, and indicates the number (or the height of a tone) of the key 11 of the electronic musical instrument 2. Further, as an example of determining whether or not the set condition is satisfied, it may be determined whether or not a tune in performance is transposed. As an example of determining whether or not the set condition is satisfied, it may be determined whether or not the tempo (rhythm) of a tune during a musical performance has changed (sudden change in tempo (tempo), sudden change in the average value of the latest strength values, change in tempo, and detection of breaks). As an example of determining whether or not the set condition is satisfied, it may be determined whether or not the user has depressed a specific pedal, not shown, attached to the electronic musical instrument 2. Further, as an example of determining whether or not the set condition is satisfied, it may be determined whether or not a phrase in performance is raised by a semitone compared to an immediately preceding phrase.

《S1005》

When it is determined that the performance data satisfies the set condition (yes in S1004), the control unit 105 (the image determination unit 1052 and the image information output unit 1053) of the information processing device 1 changes the drawing setting of the live image. Specifically, when the control unit 105 (the image determining unit 1052 and the image information output unit 1053) of the information processing device 1 generates a live image in accordance with information that can be determined based on performance data acquired after acquisition of performance data determined to satisfy the set condition, the image group is switched and any image data included in a different image group is displayed.

《S1006》

The control unit 105 (performance end determination unit 1055) of the information processing device 1 performs the performance end determination process when the communication unit 103 of the information processing device 1 does not receive the performance information transmitted from the electronic musical instrument 2 (no in step S1001), when it is determined that the performance data does not satisfy the set conditions (no in step S1004), or after step S1005. Specifically, the control unit 105 of the information processing device 1 determines whether or not a predetermined time has elapsed without input of performance information. When a predetermined time has not elapsed without the performance information being input (no in step S1006), the control unit 105 of the information processing device 1 proceeds to step S1001 again to determine whether or not performance information is input.

《S1007》

When the predetermined time has elapsed without the input of the performance information (yes in step S1006), that is, when the reception of the performance information has not been detected for the predetermined time, the control unit 105 (performance end determination unit 1055) of the information processing device 1 determines that the performance has ended. When it is determined that the performance is completed, the control unit 105 (the image determining unit 1052 and the image information output unit 1053) of the information processing device 1 proceeds to step S1007, and performs post-performance image output processing for outputting and displaying a post-performance image on the display device 4, thereby completing the entire processing.

In the case where the present embodiment is applied to automatically reproduce accompaniment data and specify a melody (melody) to the player, it may be determined that the performance is finished at the time point when the automatic reproduction of the accompaniment data is finished, and the post-performance image output process may be performed. That is, the reason why the control unit 105 of the information processing device 1 determines the end of the performance by the player may be any, and the post-performance image output process may be executed at the time point when the control unit 105 of the information processing device 1 determines the end of the performance by the player.

On the other hand, when the predetermined time has not elapsed without the performance information being input (no in step S1006), the control unit 105 of the information processing device 1 proceeds to step S1001 again to determine whether or not the performance information is input.

< 1-2-2 > specific examples

Next, a specific example of the operation of the information processing device 1 will be described.

In the specific example, for example, a case where the user plays "beautiful (fur chess)" and switches the image group at an arbitrary phrase of a tune will be described.

< 1-2-2-1 > phrase

A phrase for switching the image group by the user will be described with reference to fig. 10. Fig. 10 shows a spectral plane played by a user.

For example, in a case where the user plays "alice-causing", in a case where the image group is switched at the 3 rd phrase shown in fig. 10, a phrase immediately before the 3 rd phrase is registered as the 2 nd phrase in the information processing apparatus 1. Note that "registration" means the same meaning as "storage". Note that the phrase before the phrase 2 is denoted as phrase 1.

< 1-2-2-2 > registration method

Next, a phrase registration method will be described with reference to fig. 11. Fig. 11 is a flowchart showing an example of a method of registering "set conditions" for determination by the control unit 105 of the information processing apparatus 1 in step S1004 of fig. 9. In order for the user to switch the image group at the 3 rd phrase shown in fig. 10, the 2 nd phrase needs to be registered with the information processing apparatus 1.

《S2001》

The user selects a function of drawing a setting change phrase registration by, for example, pressing a registration button (or a registration icon) from a menu screen of an application displayed on the display unit 102 of the information processing apparatus 1. The function of registering the drawing setting change phrase is executed by the condition registration unit 1056, for example.

《S2002》

When the function of drawing setting change phrase registration is selected, the control unit 105 of the information processing apparatus 1 determines whether or not the performance information transmitted from the electronic musical instrument 2 is received by the communication unit 103 of the information processing apparatus 1 and the performance information is input (step S2002). If no performance information is input (no in step S2002), the control unit 105 of the information processing device 1 proceeds to step S2004.

《S2003》

When the performance information is input (yes in step S2002), the control unit 105 of the information processing device 1 stores the performance information in the storage unit 106 (for example, the RAM1061) of the information processing device 1. As the performance information stored in the RAM1061, for example, "note (or note number)" is given.

《S2004》

The control unit 105 of the information processing device 1 determines whether "a predetermined time has elapsed without musical performance information" or "the maximum number of notes has been reached". The maximum note number is the maximum value of the number of notes that can be stored as a registered phrase, and can be changed as appropriate. The term "phrase" is a general term of phrases registered by the user.

When the control unit 105 of the information processing device 1 determines that "the predetermined time has elapsed without the musical performance information" or "the maximum number of notes has not been reached" (no in S2004), it repeats step S2002.

《S2005》

When determining that "the predetermined time has elapsed without the performance information" or "the maximum number of notes has been reached" (yes in S2004), the control unit 105 of the information processing apparatus 1 registers (stores) the plurality of notes stored in the RAM1061 as "phrases" in the RAM 1061.

Also, for example, the user selects an image group, and the control section 105 of the information processing apparatus 1 associates "phrase" registered in the RAM1061 with the image setting file. In other words, the control unit 105 of the information processing apparatus 1 specifies the image group when the "phrase" is detected. In addition, the associated information of the "phrase" and the image group is stored in the RAM1061, for example.

< 1-2-2-3 > phrase detection method

Next, a phrase detection method will be described with reference to fig. 12. Fig. 12 is a flowchart showing a specific example of the determination method of the "set condition" in step S1004 of fig. 9. The detection of the phrase is performed in step S1004 shown in fig. 9.

In step S1004 shown in fig. 9, the control unit 105 of the information processing device 1 determines whether or not the performance data satisfies the set conditions. In this specific example, when it is determined whether or not the set condition is satisfied, it is determined whether or not the set phrase is played based on the performance data. A specific flowchart in the case of determining whether or not to perform a set phrase from performance data will be described below.

《S3001》

The control unit 105 of the information processing apparatus 1 determines whether or not the note included in the inputted performance information matches the note of the 1st note of the registered phrase stored in the RAM 1061.

《S3002》

When determining that the note included in the inputted performance information matches the note of the 1st note of the registered phrase stored in the RAM1061 (yes in S3001), the control unit 105 of the information processing apparatus 1 determines whether or not the note of the registered phrase stored in the RAM1061 has ended.

《S3003》

When determining that the note of the registered phrase stored in the RAM1061 has ended (yes in S3002), the control unit 105 of the information processing device 1 determines that the inputted musical performance information is the registered phrase. That is, the control unit 105 of the information processing apparatus 1 detects "phrase".

Thus, in step S1004 shown in fig. 9, the control unit 105 of the information processing device 1 determines that the performance data satisfies the set condition (yes in S1004).

《S3004》

When determining that the note of the registered phrase stored in the RAM1061 has not ended (S3002: no), the control unit 105 of the information processing device 1 calculates an error (difference in note number) between the inputted musical performance information (note) and the note of the registered phrase. The control unit 105 of the information processing device 1 integrates the error every time step S3004 is performed. Such an accumulation is written as an accumulation error, for example.

《S3005》

The control unit 105 of the information processing device 1 determines whether or not the integrated error calculated in step S3004 is larger than a threshold (integrated error > threshold). The threshold value is stored in the RAM1061 or the ROM1062, for example, and can be set as appropriate.

For example, there are cases where the phrase played by the user is not the registered phrase but happens to include the same sound as the registered phrase, or cases where the user has missed the registered phrase while playing it. In the case where the phrase played by the user is not a registered phrase, switching of the image group is not desirable because it is not an action desired by the user. Therefore, the control section 105 of the information processing apparatus 1 needs to appropriately determine that the phrase played by the user is not the registered phrase in step S3005. Further, it is hard for the user to determine not to register a phrase only by mistake although the user is playing the registered phrase. One of the original purposes of the performance data display system is to assist the user in practicing musical instruments. Therefore, it is necessary to assume a case where the user cannot properly play the registered phrase. Therefore, the control section 105 of the information processing apparatus 1 needs to appropriately determine that the user view played the registered phrase in step S3005.

《S3006》

If the control unit 105 of the information processing device 1 does not determine that the integrated error is larger than the threshold (no in S3005), it selects a note subsequent to the registered phrase. Then, the process proceeds to step S3002.

《S3007》

When the control unit 105 of the information processing device 1 determines that the integrated error is larger than the threshold (yes in S3005), it determines that the input performance information is not a registered phrase.

Thus, in step S1004 shown in fig. 9, the control unit 105 of the information processing device 1 determines that the performance data does not satisfy the set condition (no in S1004). That is, the control section 105 of the information processing apparatus 1 cannot detect "phrase".

< 1-2-2-4 > example of rendering of real-time image

Next, with reference to fig. 13 and 14, a case will be described in which the user plays the 2 nd phrase in fig. 10 and then plays the 3 rd phrase, and the image group is switched. Fig. 13 shows a real-time image in the case where the user plays the 2 nd phrase in fig. 10. Fig. 14 shows a real-time image in the case where the user plays phrase 3 in fig. 10. Here, it is assumed that the 2 nd phrase shown in fig. 10 is registered in the RAM1061 by the action shown in fig. 11. Then, the control unit 105 of the information processing apparatus 1 performs the operations shown in fig. 9 and 12. Here, an example in which the 1st image group is switched to the 2 nd image group by playing the registration phrase will be described as an example.

As shown in fig. 13, when the user plays phrase 2 in fig. 10, the control unit 105 of the information processing apparatus 1 causes the display apparatus 4 to display images (characters of insects) included in the 1st image group as live images (see B1 to B10 in fig. 1). At this time, an image (the character of the worm) is displayed, for example, along arrow a 1. The control unit 105 of the information processing device 1 performs the operation described with reference to fig. 12 while displaying the live image, and determines that phrase 2 is the registered phrase.

By playing the 2 nd phrase by the user, the control section 105 of the information processing apparatus 1 switches the image group from the 1st image group to the 2 nd image group. As shown in fig. 14, when the user plays phrase 3 in fig. 10, the control unit 105 of the information processing apparatus 1 causes the display apparatus 4 to display images (characters of animals) included in the 2 nd image group as live images (see C1 to C11 in fig. 2). At this time, for example, an image (character of an animal) is displayed along arrow a 2.

In this way, the user registers the phrase for switching the image group in the information processing apparatus 1 in advance, and the information processing apparatus 1 recognizes the registered phrase to switch the image group. Therefore, the user can display an arbitrary image (character) at an arbitrary timing.

< 1-2-2-5 > example of drawing of image after performance

Next, a post-performance image after the user finishes the performance of the tune shown in fig. 10 will be described with reference to fig. 15. Fig. 15 shows a post-performance image after the user finishes the performance of the tune shown in fig. 10. Here, as in the case of the example described in fig. 13 and 14, an example in which the 1st image group is switched to the 2 nd image group by playing the registration phrase will be described as an example.

When the control unit 105 of the information processing device 1 determines that the user has finished the performance, it causes the display device 4 to display the post-performance image. As shown in fig. 15, the control unit 105 of the information processing apparatus 1 arranges the image of the 1st group (the character of the worm) at the position corresponding to the 2 nd phrase (see B1 to B10 in the figure), and arranges the image of the 2 nd group (the character of the animal) at the position corresponding to the 3 rd phrase (see C1 to C11 in the figure).

This can provide the same effects as those described above.

< 1-3 > Effect

According to the above-described embodiment, in the performance data display system in which the images are drawn in accordance with the performance (music) of the user, when it is determined that the performance data satisfies the set condition, the image group used in the drawn image is switched.

For example, a performance data display system that depicts images corresponding to a user's performance (music) can stimulate the exercise will of the user's electronic musical instrument. Therefore, by adopting the above-described embodiment as well, it is possible to change the drawing setting without performing an operation not related to the performance for the drawing. Therefore, the range of the rendering expression can be expanded without performing the unique operation not related to the performance for the rendering, and the expression change can be reliably presented. As a result, it is possible to concentrate the user on the performance and improve the expressiveness of the image drawn by the performance. This can further stimulate the exercise will of the electronic musical instrument of the user.

<2> others

In the above-described embodiment, the control unit 105 of the information processing apparatus 1 switches the image group when the drawing setting is changed in step S1005 in fig. 9. However, the present invention is not limited to this, and examples thereof may include "change in brightness", "change in size of an image (character)", "change in background color (color of background of a live image or an image after playing)", change in color table of an image (character) "," change in type of an image (bonus character) to be displayed when a specific condition is satisfied ", and" change in motion of an image (character) ". In addition, a background color (color of the background of the real-time image or the image after performance), a color table of the image (character) is stored in the ROM1062, for example. The image (bonus character) displayed when the specific condition is satisfied is, for example, an image displayed on a live image when the control unit 105 of the information processing device 1 determines that the tempo of the user's performance or the like is stable. Further, the image (character) may be moved by animation or the like. The same image (character) may be included in a plurality of image groups.

In the above-described embodiment, the control unit 105 of the information processing device 1 may determine whether or not a tune is transferred during a musical performance in step S1004 of fig. 9. In this case, since the accuracy of tone determination is low immediately after the start of the performance, the drawing setting based on the detected tone may not be changed for a certain period of time from the start of the performance.

Several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments may be implemented in other various forms, and various omissions, substitutions, and changes may be made without departing from the spirit of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

Claims

1. A display method, characterized in that,

The following processing is performed by the processor of the information processing device:

displaying the first image data corresponding to the information that can be determined from the acquired first performance data;

determining whether the second performance data acquired after the acquisition of the first performance data satisfies the set condition;

When it is determined that the conditions set above are satisfied, even if the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the same as the information that can be determined based on the first performance data. Second image data in a different form from the first image data described above is displayed.

2. The display method according to claim 1, wherein,

The above-mentioned first image data is included in the first image group;

The above-mentioned second image data is included in the second image group;

The same image data may be included in the first image group and the second image group.

3. The display method according to claim 1 or 2, characterized in that,

The conditions that are determined to satisfy the conditions set above include at least one of the following conditions, namely:

It is determined based on the acquired second performance data that the performance of the set phrase has been performed;

If it is determined to be transferred; and

It is determined that the rhythm has changed.

4. The display method according to any one of claims 1 to 3, wherein,

The above-mentioned determinable information includes at least one of a note name and a chord type.

5. The display method according to any one of claims 1 to 4, wherein,

Determine whether the performance is over;

If it is not determined that the performance is over, display the image data according to the set first condition;

When it is determined that the performance has ended, the image data is displayed according to the set second condition.

6. An electronic device, characterized in that,

including a display device and a processor;

The above processor performs the following processing:

Instructing the display device to display the first image data corresponding to the information that can be determined based on the acquired first performance data;

When it is determined that the conditions set above are satisfied, even if the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the same as the information that can be determined based on the first performance data. The display device is instructed to display second image data in a form different from that of the first image data.

7. The electronic device of claim 6, wherein,

The above-mentioned first image data is included in the first image group;

The above-mentioned second image data is included in the second image group;

8. The electronic device according to claim 6 or 7, characterized in that,

If it is determined to be transferred; and

It is determined that the rhythm has changed.

9. The electronic device according to any one of claims 6 to 8, characterized in that,

10 . The electronic device according to claim 6 , wherein: 10 .

Determine whether the performance is over;

If it is not determined that the performance is over, instructing the display device to display image data according to the set first condition;

When it is determined that the performance has ended, the display device is instructed to display the image data in accordance with the set second condition.

11. A performance data display system, characterized in that,

A terminal device having an electronic musical instrument and a display device;

The above electronic musical instrument is processed as follows:

generating first performance data corresponding to the user's first performance operation, and outputting the generated first performance data to the display device;

generating second performance data corresponding to the user's second performance operation, and outputting the generated second performance data to the display device;

generating third performance data corresponding to the third performance operation of the user, and outputting the generated third performance data to the display device;

The above-mentioned terminal device performs the following processing:

determining whether the second performance data acquired after the acquisition of the first performance data satisfies a set condition;

When it is determined that the conditions set above are satisfied, even if the information that can be determined based on the third performance data acquired after the acquisition of the second performance data is the same as the information that can be determined based on the first performance data, Second image data in a form different from the above-described first image data is also displayed.

12. A storage medium storing a program, characterized in that,

The above program causes the computer of the information processing apparatus to perform the following processing: