JP2024121044A - PROGRAM, INFORMATION PROCESSING APPARATUS AND IMAGE GENERATION METHOD - Google Patents

Abstract

[Problem] To make it possible to capture and visualize the impression given by a performance. [Solution] A control unit of an information processing device calculates an evaluation value for judging the impression given by a performance from performance information based on the performance, and synthesizes a pattern image that visually gives the impression given by the performance based on the calculated evaluation value with an image and outputs the result. For example, the control unit 11 synthesizes a performance image based on a feature amount acquired by analyzing the performance information with a pattern image that visually gives the impression given by the performance based on the evaluation value and outputs the result. [Selected Figure] Figure 3

Description

The present invention relates to a program, an information processing device, and an image generation method.

Conventionally, karaoke devices are known that select the next song to be played, the background image, and the lyric image based on the results of a singing ability assessment (see, for example, Patent Document 1).

Japanese Patent Application Publication No. 6-110479

However, the technology described in Patent Document 1 cannot capture and visualize the impression given by the performance while it is being played.

The present invention was made in consideration of the above problems, and aims to make it possible to capture and visualize the impression given by a performance.

In order to solve the above problems, the program of the present invention comprises:
On the computer,
calculating an evaluation value for determining an impression given by the performance from performance information based on the performance;
a pattern image that visually conveys an impression of the performance based on the calculated evaluation value is synthesized with the image and output.
Execute the process.

The present invention makes it possible to visualize the impression that a performance gives.

1 is a diagram showing an example of the overall configuration of a performance system including an information processing device according to the present invention; FIG. 13 is a diagram showing an example of a pattern image (background image). 2 is a flowchart showing the flow of a performance image generating process A executed by the control unit of FIG. 1 . 13A to 13C are diagrams showing examples of characters etc. used in the first image. 4 is a flowchart showing the flow of a background selection process executed in step S4 of FIG. 3. FIG. 13 is a diagram showing an example of a first image onto which a pattern image (background image) is combined. FIG. 13 is a diagram showing an example of a mask image. 13 is a flowchart showing the flow of a performance image generating process B executed by the control unit of FIG. 1 . 10 is a flowchart showing the process of mask selection executed in step S24 of FIG. 8; FIG. 13 is a diagram showing an example of a first image combined with a mask image.

The following describes an embodiment of the present invention with reference to the drawings. However, the embodiment described below has various limitations that are technically preferable for implementing the present invention. Therefore, the technical scope of the present invention is not limited to the following embodiment and illustrated examples.

[Configuration of the performance system 100]
First Embodiment
FIG. 1 is a diagram showing an example of the overall configuration of a performance system 100 according to the present invention.
1, the performance system 100 includes an information processing device 1 and an electronic musical instrument 2. The information processing device 1 and the electronic musical instrument 2 are communicatively connected to each other wirelessly or via a wire so as to be able to transmit and receive data.

[Configuration of information processing device 1]
The information processing device 1 is a device that generates an image in response to a performance of an electronic musical instrument 2 by a user (player).
The information processing device 1 may be, for example, a tablet PC (Personal Computer), a notebook PC, a smartphone, or the like.

As shown in FIG. 1, the information processing device 1 is configured with a control unit 11, a memory unit 12, an input unit 13, a display unit 14, and a communication unit 15, and each unit is connected by a bus 16.

The control unit 11 is a computer that includes at least one processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like, and controls each part of the information processing device 1. Specifically, the CPU of the control unit 11 reads out a specified program from among the system programs and various application programs stored in the storage unit 12, expands it in the RAM, and executes various processes in cooperation with the expanded program.
For example, when the input unit 13 is operated to instruct the launch of a performance image generation application 121 stored in the memory unit 12, the CPU of the control unit 11 launches the performance image generation application 121 and executes the performance image generation process A (see Figure 3) described later.

The storage unit 12 is composed of a non-volatile semiconductor memory, a HDD (Hard Disk Drive), etc. The storage unit 12 stores the system program of the information processing device 1, various application programs, and data required for executing the programs. The storage unit 12 is not limited to being built into the information processing device 1, and may include an external recording medium that is detachable from the information processing device 1, such as an external HDD or a USB memory.
In this embodiment, the storage unit 12 stores a performance image generating application 121 for the control unit 11 to execute the performance image generating process A.

2, the storage unit 12 stores pattern images (texture images) that visually give impressions of "delicate,""powerful,""slow,""fast,""sad,""joyful,""heavy,""light,""regular," and "free," in association with pre-assigned background numbers. The pattern images stored in the storage unit 12 are background images that become the background of the performance image (described in detail later).
In this embodiment, the pattern image that visually gives the impression of "delicate" is associated with the background number "1". The pattern image that visually gives the impression of "powerful" is associated with the background number "2". The pattern image that visually gives the impression of "slow" is associated with the background number "3". The pattern image that visually gives the impression of "fast" is associated with the background number "4". The pattern image that visually gives the impression of "sad" is associated with the background number "5". The pattern image that visually gives the impression of "fun" is associated with the background number "6". The pattern image that visually gives the impression of "heavy" is associated with the background number "7". The pattern image that visually gives the impression of "light" is associated with the background number "8". The pattern image that visually gives the impression of "regular" is associated with the background number "9". The pattern image that visually gives the impression of "free" is associated with the background number "10".

The input unit 13 is composed of push button switches, a touch panel attached to the display unit 14, etc. The input unit 13 detects the operation of the push button switches or touch operations on the screen by the user, and outputs an operation signal to the control unit 11.

The display unit 14 is composed of an LCD (Liquid Crystal Display), an EL (Electro Luminescence) display, etc., and performs various displays according to the display information instructed by the control unit 11.

The communication unit 15 includes a wired unit or a wireless unit for communicating with external devices such as the electronic musical instrument 2, and transmits and receives data to and from the external devices. In this embodiment, the communication unit 15 transmits and receives data to and from the electronic musical instrument 2 via a communication interface (in this embodiment, a MIDI interface) not shown.

[Electronic Musical Instrument 2]
The electronic musical instrument 2 is, for example, a keyboard instrument.
As shown in FIG. 1, the electronic musical instrument 2 is configured with a control unit 21, a memory unit 22, a keyboard 23, an input unit 24, a display unit 25, a communication unit 26, an output unit 27, etc., and each unit is connected by a bus 28.

The control unit 21 is a computer that includes various processors such as a CPU and a DSP (Digital Signal Processor), a RAM, etc., and controls the various components of the electronic musical instrument 2. Specifically, the CPU of the control unit 21 reads out a specified program from among the system programs and various programs stored in the storage unit 22, loads it into the RAM, and executes various processes in cooperation with the loaded program.
For example, the control unit 21 causes the output unit 27 to output or mute instrument sounds in accordance with a performance operation (in this embodiment, key pressing/release) on the keyboard 23. Furthermore, when the control unit 21 detects a performance operation on the keyboard 23, it generates performance information in accordance with the performance operation, i.e., based on the performance, and transmits the performance information to the information processing device 1 via the communication unit 26. The performance information includes information on the pitch (note number), velocity and timing of the pressed key, and the pitch and timing of the released key.

The storage unit 22 is composed of a non-volatile semiconductor memory, a HDD (Hard Disk Drive), etc. The storage unit 22 stores the system program of the electronic musical instrument 2, various programs, and data required for executing the programs. The storage unit 22 is not limited to being built into the electronic musical instrument 2, but may include an external recording medium that is detachable from the electronic musical instrument 2, such as an external HDD or USB memory.

The keyboard 23 includes a plurality of keys (operators) and a detection unit that detects pressed/released keys, and outputs information such as the pitch, timing, and velocity of the keys pressed/released by the user to the control unit 21.

The input unit 24 is composed of push button switches, a touch panel attached to the display unit 25, etc. The input unit 24 detects the operation of the push button switches or touch operations on the screen by the user, and outputs an operation signal to the control unit 21.

The display unit 25 is composed of an LCD (Liquid Crystal Display) or the like, and displays various information according to the display information instructed by the control unit 21.

The communication unit 26 includes a wired unit or a wireless unit for communicating with external devices such as the information processing device 1, and transmits and receives data to and from the external devices. In this embodiment, the communication unit 26 transmits and receives data to and from the information processing device 1 via a communication interface (in this embodiment, a MIDI interface) not shown.

The output unit 27 includes a sound source unit, a D/A converter, an amplifier, a speaker, etc. In accordance with instructions from the control unit 21, the output unit 27 reads out waveform data stored in advance in a waveform ROM provided in the sound source unit or generates waveform data, and outputs musical instrument sounds based on the waveform data from the speaker via the D/A converter and amplifier.

[Operation of information processing device 1]
Next, the operation of the information processing device 1 in this embodiment will be described.
3 is a flowchart showing the flow of a performance image generating process A executed by the control unit 11 and the performance image generating application 121 in cooperation with each other when the performance image generating application 121 is started in the information processing device 1. Hereinafter, the performance image generating process A will be described with reference to FIG.

First, the control unit 11 judges whether or not performance information has been input from the electronic musical instrument 2 via the communication unit 15 (step S1).
When it is determined that no performance information has been input from the electronic musical instrument 2 via the communication section 15 (step S1; NO), the control section 11 proceeds to step S8.

When it is determined that performance information has been input from the electronic musical instrument 2 via the communication section 15 (step S1; YES), the control section 11 acquires the input performance information and analyzes the performance information (step S2).
In step S2, for example, the control unit 11 analyzes the input performance information to obtain at least one of the features of the music being played, such as pitch, velocity, key (for example, 24 types from C major to B minor), tempo, chord type (for example, major, minor, sus, arg, dim, 7th, etc.), and number of chords. The pitch and velocity can be obtained from the input performance information. The key and chord type are not particularly limited, but can be obtained using a method disclosed in Japanese Patent No. 3211839, etc. The tempo is not particularly limited, but can be obtained using a method disclosed in Japanese Patent No. 5672280, etc. The number of chords is not particularly limited, but can be obtained based on the number of simultaneously pressed keys (the number of keys pressed within a predetermined time), etc.
However, the feature amount obtained in step S2 is not limited to the above, and for example, the presence or absence of syncopation, the regularity of beats, chord progression, etc. may be obtained.

Next, the control unit 11 generates a first image based on the analysis result in step S2 (step S3).
Here, the first image is an image (performance image) corresponding to a real-time performance, which is generated each time performance information is input. In this embodiment, the first image is generated by processing a base image stored in the storage unit 12 using computer graphics (CG), but the first image does not need to be generated by such a method and may be any image corresponding to the performance information.

FIG. 4 is a diagram showing examples of characters and the like used in the first image.
As shown in Fig. 4(a), 12 types of flower images corresponding to pitches are stored in the storage unit 12 as character images corresponding to melodies. That is, flower images are stored in association with pitches in the storage unit 12. Note that Fig. 4(a) shows three of the 12 types of flower images.
The control unit 11 selects an image of a flower that corresponds to the pitch of the melody obtained from the performance information as a character to be used in the first image.

As shown in Fig. 4(b), 10 types of leaf images corresponding to the code types (e.g., major, minor, sus, arg, dim, 7th, etc.) are stored in the storage unit 12 as character images corresponding to the code types. That is, the storage unit 12 stores leaf images in association with the code types. Note that Fig. 4(b) shows three of the 10 types of leaf images.
The control unit 11 selects an image of a leaf that corresponds to the chord type acquired based on the performance information as a character to be used in the first image.

The control unit 11 also selects the size of the character used in the first image based on the velocity obtained from the performance information. For example, as shown in the image on the left side of FIG. 4(c), when the velocity is low (the velocity is within the first range), the control unit 11 reduces the size of the image of the flower as a character, and as the velocity moves into the second range, which is a range of values greater than the maximum value of the first range, and into the third range, which is a range of values greater than the maximum value of the second range, the control unit 11 gradually increases the size of the image of the flower as a character, as shown in the image on the center side and the image on the right side of FIG. 4(c). The same applies to the image of the leaf.

In addition, when the control unit 11 acquires information in step S2 that there is syncopation and the regularity of the beats is broken, as the degree of the break increases, the image on the left side of Figure 4 (d) gradually changes to the image on the right side, that is, the shape of the leaf image as a character becomes more broken.

When the control unit 11 acquires information on the regularity of the beats in the performance information and the correctness of the chord progression in step S2, it selects whether or not to color the characters used in the first image (whether to use a line drawing or color), etc., according to this information, as shown in FIG. 4(e). For example, if the regularity of the beats or the correctness of the chord progression is poor, the image of the leaf as the character will be a poorly colored line drawing (left image), and conversely, if it is good, it will be selected to be colored (right image).

The control unit 11 may also select the background color based on the key. Colors corresponding to the key have been proposed, for example, by Alexander Scriabin.

Then, as shown in Figure 4(f), the control unit 11 determines the position of the character in the first image so that the character appears on the display unit 14 according to one of 14 types of trajectory patterns PS (three of which are shown in Figure 4(f)) corresponding to the chord progression, and generates a first image in which the character is applied to match that position.
Each time the control unit 11 generates a first image, it stores the image parameters used to generate the first image (e.g., the identification number of the selected character (flower, leaf), size, whether the character has color, background color, character placement, etc.) in the RAM or memory unit 12.

Next, the control unit 11 executes a background selection process (step S4).
5 is a flow chart showing the flow of the background selection process. The background selection process is executed by the control unit 11 and the musical performance image generation application 121 in cooperation with each other.

First, the control unit 11 calculates a Semantic Differential (SD) vector (step S401).
Here, the SD vector is composed of one or more evaluation values (SD values) calculated to determine the impression given by the performance based on the performance information using the SD (Semantic Differential) method. Each of the one or more SD values is an evaluation value for determining whether the performance information based on the performance gives one impression of a predetermined antonym pair (a pair of words having opposite meanings), the other impression, or neither impression, and is calculated based on the feature amount acquired in step S2.
In this embodiment, the control unit 11 calculates five SD values for five antonym pairs, "delicate-powerful,""slow-fast,""sad-happy,""heavy-light," and "regular-free," to determine whether the pair gives one impression, the other impression, or neither impression.
For example, the SD value for "delicate-powerful" (called SD1) is calculated based on the average velocity over a predetermined period (eg, the time equivalent to the most recent beat).
Also, for example, the SD value for "slow-fast" (called SD2) is calculated based on the average tempo over a predetermined period.
Also, for example, the SD value for "sad-happy" (called SD3) is calculated based on the average of the tonality (a value obtained by converting the tonality into a numerical value) and the tempo over a predetermined period.
Also, for example, the SD value for "heavy-light" (called SD4) is calculated based on the average pitch, number of chords, and tempo over a predetermined period.
Also, for example, the SD value for "regular-free" (called SD5) is calculated based on the variance of the tempo over a given period of time.
The control unit 11 stores the calculated SD values (SD1 to SD5) in the RAM or the storage unit 12.

In this embodiment, each SD value ranges from 0 to 255. The control unit 11 calculates the SD value by converting the possible value range of each feature amount acquired in step S2 so that the minimum value is 0 and the maximum value is 255.
The SD value approaches 0 as the impression of one of the antonym pair is stronger (more prominent), and approaches 255 as the impression of the other is stronger (more prominent). For example, when the antonym pair is "delicate-powerful", the more delicate the performance, the closer SD1 is to 0, and the more powerful the performance, the closer SD1 is to 255. When the SD value is less than the first threshold, the control unit 11 determines that the performance gives the impression of one of the antonym pair. When the SD value exceeds a second threshold that is greater than the first threshold, the control unit 11 determines that the performance gives the impression of the other. When the SD value is equal to or greater than the first threshold and equal to or less than the second threshold, the control unit 11 determines that the performance gives neither impression (first threshold<second threshold). In this embodiment, the first threshold is 30, and the second threshold is 225.

Next, the control unit 11 sets the "background" variable, which stores the background number, to "-1" (step S402).

Next, the control unit 11 judges whether SD1<30 (step S403).
If it is determined that SD1<30 (step S403; YES), the control unit 11 determines that the performance gives a "delicate" impression, sets the "background" variable to "1" (step S404), and proceeds to step S5 in FIG.

When it is determined that SD1<30 is not satisfied (step S403; NO), the control unit 11 determines whether or not SD1>225 is satisfied (step S405).
If it is determined that SD1>225 (step S405; YES), the control unit 11 determines that the performance gives a "powerful" impression, sets the "background" variable to "2" (step S406), and proceeds to step S5 in FIG.

If it is determined that SD1>225 is not true (step S405; NO), that is, if the performance does not give the impression of being either "delicate" or "powerful", the control unit 11 determines whether SD2<30 is true (step S407).
If it is determined that SD2<30 (step S407; YES), the control unit 11 determines that the performance gives the impression of being "slow," sets the "background" variable to "3" (step S408), and proceeds to step S5 in FIG.

When it is determined that SD2<30 is not satisfied (step S407; NO), the control unit 11 determines whether or not SD2>225 is satisfied (step S409).
If it is determined that SD2>225 (step S409; YES), the control unit 11 determines that the performance gives the impression of being "fast," sets the "background" variable to "4" (step S410), and proceeds to step S5 in FIG.

If it is determined that SD2>225 is not true (step S409; NO), that is, if the performance does not give the impression of being "slow" or "fast," the control unit 11 determines whether SD3<30 is true (step S411).
If it is determined that SD3<30 (step S411; YES), the control unit 11 determines that the performance gives a "sad" impression, sets the "background" variable to "5" (step S412), and proceeds to step S5 in FIG.

When it is determined that SD3<30 is not satisfied (step S411; NO), the control unit 11 determines whether or not SD3>225 is satisfied (step S413).
If it is determined that SD3>225 (step S413; YES), the control unit 11 determines that the performance gives a "fun" impression, sets the "background" variable to "6" (step S414), and proceeds to step S5 in FIG.

If it is determined that SD3 is not greater than 225 (step S413; NO), that is, if the performance does not give a "sad" impression or a "happy" impression, the control unit 11 determines whether SD4 is less than 30 (step S415).
If it is determined that SD4<30 (step S415; YES), the control unit 11 determines that the performance gives a "heavy" impression, sets the "background" variable to "7" (step S416), and proceeds to step S5 in FIG.

When it is determined that SD4 is not smaller than 30 (step S415; NO), the control unit 11 determines whether or not SD4 is larger than 225 (step S417).
If it is determined that SD4>225 (step S417; YES), the control unit 11 determines that the performance gives a "light" impression, sets the "background" variable to "8" (step S418), and proceeds to step S5 in FIG.

If it is determined that SD4 is not greater than 225 (step S417; NO), that is, if the performance does not give the impression of being either "heavy" or "light," the control unit 11 determines whether SD5 is less than 30 (step S419).
If it is determined that SD5<30 (step S419; YES), the control unit 11 determines that the performance gives a "regular" impression, sets the "background" variable to "9" (step S420), and proceeds to step S5 in FIG.

When it is determined that SD5<30 is not satisfied (step S420; NO), the control unit 11 determines whether or not SD5>225 is satisfied (step S421).
If it is determined that SD5>225 (step S421; YES), the control unit 11 determines that the performance gives a "free" impression, sets the "background" variable to "10" (step S422), and proceeds to step S5 in FIG.
If it is determined that SD5>225 is not satisfied (step S421; NO), that is, if the performance does not give a "regular" impression or a "free" impression, the control unit 11 proceeds to step S5 in FIG.

In step S5 of FIG. 3, the control unit 11 determines whether the value of the "background" variable is "-1" (step S5).
If it is determined that the value of the "background" variable is not "-1" (step S5; NO), the control unit 11 reads out from the memory unit 12 a pattern image to which the value of the "background" variable corresponds as a background number, and combines it with the first image as a background image (step S6), and proceeds to step S7.

FIG. 6 is a diagram showing an example of a first image in which the pattern images shown in FIG. 2 are combined.
As shown in Fig. 6, the first image generated according to the performance has a pattern image in the background synthesized therein, which visually gives a prominent impression of the performance. Each image in Fig. 6 includes not only the pattern image, which is the background, but also the first image in step S3. This is because the first image and the pattern image in step S3 are displayed according to the characteristic amount and the evaluation value of the performance, respectively, in other words, it is not the case that only the pattern image is displayed.
Here, the first image is an image in which flower and leaf characters corresponding to pitches and chord types are drawn in a size corresponding to the velocity and in a background color corresponding to the key, as described above. In this way, the first image is an image generated according to the performance, but it is difficult for the user to recognize from this image what kind of impression the performance has. In particular, the longer the performance, the more characters there are in the first image, making it look more complicated. On the other hand, in the first image of this embodiment, as shown in FIG. 6, a pattern image that visually gives a noticeable impression of the performance is synthesized in the background. Therefore, the impression of the performance can be more clearly conveyed to the user.

On the other hand, if it is determined that the value of the "background" variable is "-1" (step S5; YES), that is, if the performance does not give a significant impression and a background pattern image has not been selected, the control unit 11 proceeds to step S7 without combining the background with the first image. Note that, if it is determined in step S421 that SD5 is not greater than 225, an intermediate background image that is not suitable for any of the attributes "delicate," "powerful," "slow," "fast," "sad," "fun," "heavy," "light," "regular," and "free" may be applied as a replacement for the pattern image.

In step S7, the control unit 11 outputs the first image (step S7), and the process proceeds to step S8.
For example, the control unit 11 outputs the first image to the display unit 14 to display it. Furthermore, the control unit 11 may output the first image to a projector (not shown) or the like via the communication unit 15 to display (project) the first image using the projector.

In step S8, the control unit 11 judges whether or not a predetermined time has elapsed without inputting any performance information (step S8).
When it is determined that the predetermined time has not elapsed without input of performance information (step S8; NO), the control section 11 returns to step S1.

If it is determined that a predetermined time has elapsed without any performance information being input (step S8; YES), the control unit 11 determines that the performance has ended and generates a second image summarizing the entire performance (step S9).
The control unit 11 generates the second image based on a group of image parameters used to generate the multiple first images generated during the performance.

Next, the control unit 11 executes a background selection process (step S10).
In step S10, a background image for the second image is selected. The background selection process in step S10 is as shown in the flowchart of FIG. 5, but in step S401, SD1 to SD5 to be used in selecting the background image for the second image are calculated based on SD1 to SD5 stored during performance. For example, the average values of SD1 to SD5 stored during performance are calculated as SD1 to SD5 to be used in selecting the background image for the second image. Then, the background image for the second image is selected by the processes of steps S402 to S422.

Returning to FIG. 3, when the background image selection process is completed, the control unit 11 judges whether the value of the "background" variable is "-1" (step S11).
If it is determined that the value of the "background" variable is not "-1" (step S11; NO), the control unit 11 reads out a pattern image whose value of the "background" variable is the background number from the memory unit 12, combines it with the second image as a background image (step S12), and proceeds to step S13.

The second image generated in step S12 has a pattern image synthesized in the background that visually conveys a prominent impression given by the overall performance. Therefore, the impression given by the overall performance can be more clearly conveyed to the user.

On the other hand, if the value of the "background" variable is determined to be "-1" (step S11; YES), that is, if the performance does not give a significant impression and a pattern image to serve as the background has not been selected, the control unit 11 proceeds to step S13 without overlaying the background onto the second image. Note that if it is determined in step S421 of step S10 that SD5 is not greater than 225 based on the average values of SD1 to SD5, that is, if it is determined that the performance does not fall into any of the attributes of "delicate," "powerful," "slow," "fast," "sad," "fun," "heavy," "light," "regular," and "free," an intermediate background image may be overlaid as a pattern image instead of overlaying the background onto the second image.

In step S13, the control unit 11 outputs (displays) the second image (step S13), and ends the performance image generating process A.
In step S13, the control unit 11, for example, outputs the second image to the display unit 14 to display it. Furthermore, the control unit 11 may output the second image to a projector (not shown) or the like via the communication unit 15 and cause the projector to display (project) the second image.

In this way, the control unit 11 outputs a performance image (first image and second image) during or at the end of a performance against a background of a pattern image that gives a prominent impression of the performance, allowing the user to easily recognize the impression that the performance gives.

Second Embodiment
In the second embodiment, a mask image that visually gives the impression of the performance is synthesized with a performance image (first image and second image) rather than a background image that visually gives the impression of the performance.

In the second embodiment, the performance image generation application 121 stored in the memory unit 12 includes a program for the control unit 11 to execute the performance image generation process B (see FIG. 8).

Furthermore, the memory unit 12 stores mask images (mask pattern images) that visually give the impressions of "delicate,""powerful,""slow,""fast,""sad,""joyful,""heavy,""light,""regular," and "free," in association with pre-assigned mask numbers.
7 shows examples of mask images that visually give the impressions of "delicate"-"powerful" and "sad"-"happy." Here, the mask image is an image in which the area to be masked (hidden) is specified by 0 or 1, and is a pattern image having a mask pattern (shape pattern of the mask area) that visually gives the above impressions.

In this embodiment, a mask image that visually gives the impression of "delicate" is associated with a mask number of "1". A mask image that visually gives the impression of "powerful" is associated with a mask number of "2". A mask image that visually gives the impression of "slow" is associated with a mask number of "3". A mask image that visually gives the impression of "fast" is associated with a mask number of "4". A mask image that visually gives the impression of "sad" is associated with a mask number of "5". A mask image that visually gives the impression of "fun" is associated with a mask number of "6". A mask image that visually gives the impression of "heavy" is associated with a mask number of "7". A mask image that visually gives the impression of "light" is associated with a mask number of "8". A mask image that visually gives the impression of "regular" is associated with a mask number of "9". A mask image that visually gives the impression of "free" is associated with a mask number of "10".

The rest of the configuration of the information processing device 1 is the same as that described in the first embodiment with reference to FIG. 1, so the explanation will be used to explain the operation of the second embodiment below.

Figure 8 is a flowchart showing the flow of performance image generation process B executed by the control unit 11 and the performance image generation application 121 in cooperation with each other when the performance image generation application 121 is started in the information processing device 1 in the second embodiment. Below, performance image generation process B will be described with reference to Figure 8.

In the performance image generation process B, the control unit 11 first executes the processes of steps S21 to S23. The processes of steps S21 to S23 are similar to the processes of steps S1 to S3 in FIG. 3, and therefore the explanation thereof will be used.

Next, the control unit 11 executes a mask selection process (step S24).
9 is a flowchart showing the flow of the mask selection process. The mask selection process is executed by the control unit 11 in cooperation with the performance image generating application 121 stored in the storage unit 12.

In the mask selection process, first, the control unit 11 calculates an SD vector (step S2401). The method of calculating the SD vector is the same as that described in step S401 of FIG. 5, so the description will be used here.

Next, the control unit 11 sets the "mask" variable, which is used to store the mask number, to "-1" (step S2402).

Next, the control unit 11 executes the processes of steps S2403 to S2422. The processes of steps S2403 to S2422 are the same as those of steps S403 to S422 in FIG.
When the processes of steps S2403 to S2422 are completed, the control unit 11 proceeds to step S25 in FIG.

In step S25 of FIG. 8, the control unit 11 determines whether the value of the "mask" variable is "-1" (step S25).
If it is determined that the value of the "mask" variable is not "-1" (step S25; NO), the control unit 11 reads out the mask image to which the value of the "mask" variable corresponds as a mask number from the memory unit 12, combines it with the first image (step S26), and proceeds to step S27.

FIG. 10 is a diagram showing an example of a first image in which the mask images shown in FIG. 7 are combined.
Here, as described above, the first image is an image in which flower and leaf characters associated with pitches and chord types are drawn in a size corresponding to the velocity with a background color corresponding to the key. In this way, the first image is an image generated according to a performance, but it is difficult for a user to recognize from this image what kind of impression the performance has. On the other hand, the first image of this embodiment is masked with a pattern shape that visually gives a prominent impression of the performance, as shown in FIG. 10. In other words, the first image can be output in a pattern shape that visually gives a prominent impression of the performance. Therefore, the impression of the performance can be more clearly conveyed to the user.

On the other hand, if it is determined that the value of the "mask" variable is "-1" (step S25; YES), that is, if the performance does not give a significant impression and a mask image has not been selected, the control unit 11 proceeds to step S27 without synthesizing a mask image with the first image. Note that, if it is determined in step S2421 that SD5>225 is not true, an intermediate mask image that is not suitable for any of the attributes "delicate", "powerful", "slow", "fast", "sad", "fun", "heavy", "light", "regular" and "free" may be synthesized.

In step S27, the control unit 11 outputs the first image (step S27).
For example, the control unit 11 outputs the first image to the display unit 14 for display. The control unit 11 may also output the first image to a projector or the like (not shown) via the communication unit 15 and cause the projector to display (project) the first image. When the first image combined with the mask image is projected by the projector, the unmasked portion of the second image is projected (visualized) in a shape corresponding to the prominent impression given by the performance, so that the impression given by the performance can be more clearly conveyed to the user.

Next, the control unit 11 judges whether or not a predetermined time has elapsed without input of performance information (step S28).
When it is determined that the predetermined time has not elapsed without input of performance information (step S28; NO), the control section 11 returns to step S21.

If it is determined that a predetermined time has elapsed without performance information being input (step S28; YES), the control unit 11 determines that the performance has ended and generates a second image that summarizes the entire performance (step S29).

Next, the control unit 11 executes a mask selection process (step S30).
In step S30, a mask image for the second image is selected. The mask selection process in step S30 is as shown in the flowchart of Fig. 9, but in step S2401, SD1 to SD5 to be used in selecting a mask image for the second image are calculated based on SD1 to SD5 stored during performance. For example, the average values of SD1 to SD5 stored during performance are calculated as SD1 to SD5 to be used in selecting a mask image for the second image. Then, a background image for the second image is selected by the processes of steps S2402 to S2422.

Returning to FIG. 8, when the mask image selection process is completed, the control unit 11 determines whether the value of the "mask" variable is "-1" (step S31).
If it is determined that the value of the "mask" variable is not "-1" (step S31; NO), the control unit 11 reads out the mask image whose value of the "mask" variable is the mask number from the memory unit 12 and combines it with the second image (step S32), and proceeds to step S33.

The second image generated in step S32 is synthesized with a mask image that visually conveys a prominent impression of the overall performance. Therefore, the impression of the overall performance can be more clearly conveyed to the user.

On the other hand, if it is determined that the value of the "mask" variable is "-1" (step S31; YES), that is, if the performance does not give a significant impression and a mask image has not been selected, the control unit 11 proceeds to step S33 without synthesizing a mask image with the second image. Note that if it is determined in step S2421 of step S30 that SD5 is not greater than 225 based on the average values of SD1 to SD5, that is, if it is determined that the performance does not fall into any of the attributes of "delicate," "powerful," "slow," "fast," "sad," "fun," "heavy," "light," "regular," and "free," an intermediate mask image may be synthesized instead of synthesizing a mask image with the second image.

In step S33, the control unit 11 outputs (displays) the second image (step S33), and ends the performance image generating process B.
In step S33, the control unit 11, for example, outputs the second image to the display unit 14 to display it. Alternatively, the control unit 11 may output the second image to a projector (not shown) via the communication unit 15 and cause the projector to display (project) the second image. When the second image combined with the mask image is projected by the projector, the unmasked portion of the second image is projected (visualized) in a shape corresponding to the prominent impression given by the performance, so that the impression given by the performance can be more clearly conveyed to the user.

In this way, the control unit 11 outputs a performance image (first image and second image) that is masked in a shape that gives a prominent impression of the performance during or at the end of the performance, so that the impression given by the performance can be more clearly conveyed to the user.

As described above, the control unit 11 of the information processing device 1 calculates an evaluation value for determining the impression given by the performance from performance information based on the performance, and outputs a pattern image that visually conveys the impression given by the performance based on the calculated evaluation value, combined with a performance image based on features obtained by analyzing the performance information.
Therefore, the impression given by the performance can be captured and visualized, so that the impression given by the performance can be more clearly conveyed to the user.

For example, the pattern image is a background image that serves as the background for the performance image, so the impression given by the performance can be clearly conveyed to the user by the background of the performance image.

Also, for example, the pattern image is a mask image for masking a portion of the performance image, so the impression given by the performance can be clearly conveyed to the user by the shape of the unmasked portion of the performance image.

For example, the control unit 11 calculates at least one of the following features as a characteristic amount: velocity, tempo, key, pitch, and number of chords in the performance. Therefore, an evaluation value for determining the impression given by the performance can be calculated based on at least one of the following features: velocity, tempo, key, pitch, and number of chords in the performance.

Also, for example, the control unit 11 calculates an evaluation value for determining whether the performance gives one impression of a predetermined antonym pair, the other impression, or neither impression, and if it is determined based on the calculated evaluation value that the performance gives one impression of the antonym pair, it synthesizes a first pattern image that visually gives one impression with the performance image and outputs the result.If it is determined that the performance gives the other impression of the antonym pair, it synthesizes a second pattern image that visually gives the other impression with the performance image and outputs the result.
Therefore, it is possible to grasp whether the performance gives the impression of one or the other of a predetermined antonym pair and convey this to the user.

For example, by defining antonym pairs as at least one of delicate and powerful, slow and fast, sad and happy, heavy and light, regular and free, it is possible to convey to the user whether the performance gives one or the other of these opposing impressions.

For example, the control unit 11 can calculate an evaluation value based on the average velocity over a specified period of time during a performance to determine whether the performance gives a delicate impression, a powerful impression, or neither impression.

For example, the control unit 11 can also calculate an evaluation value based on the average tempo of the performance over a specified period of time to determine whether the performance gives the impression of being slow, fast, or neither.

For example, the control unit 11 can also calculate an evaluation value for determining whether the performance gives a sad impression, a happy impression, or neither impression, based on the average key and tempo over a specified period of time during the performance.

For example, the control unit 11 can also calculate an evaluation value for determining whether a performance gives a heavy impression, a light impression, or neither, based on the average pitch, number of chords, and tempo over a specified period of time during the performance.

For example, the control unit 11 can also calculate an evaluation value based on the variance or standard deviation of the tempo during a specified period of performance to determine whether the performance gives an impression of regularity, freedom, or neither.

Note that the contents described in the above embodiment are preferred examples of the program, information processing device, and image generation method according to the present invention, and are not limited thereto.

For example, in the above embodiment, the characters used in the first and second images were flowers and leaves, but this does not need to be limited to this and other characters may be used.

In addition, in the above embodiment, an example was described in which the present invention is applied when generating a performance image (first image and second image) based on performance information corresponding to a performance operation performed by an actual performer, but the present invention may also be applied when generating a performance image based on performance information input sequentially by the automatic performance function of the electronic musical instrument 2.

In the above embodiment, the first image is generated each time performance information is input, and is output after being synthesized with a pattern image such as a background image or a mask image. However, the timing of generating and outputting the first image is not limited to this. For example, the first image may be generated and output for one or more beats, or for one or more measures. In addition, if a pattern image such as a background image or a mask image is changed and this is reflected immediately, the first image may become difficult to see, for example, in the case of a performance with many changes. Therefore, when changing the pattern image, the pattern image may be changed if a certain impression continues for a predetermined time or more after it has been determined that the impression is given.

In the above embodiment, the first threshold was set to 30 and the second threshold to 225 for all SD values, but the first and second thresholds may be different for each SD value. In the above embodiment, if all SD values are equal to or greater than the first threshold and equal to or less than the second threshold, no pattern image is synthesized. However, the first or second threshold for the last SD value to be determined may be shifted so that some pattern image is synthesized with the first or second image.

In the above embodiment, the system determines whether the SD value is smaller than the first threshold or larger than the second threshold, starting from SD1, and if neither is true, moves on to the next SD value and performs the determination. That is, in the above embodiment, SD1 has the highest priority and SD5 has the lowest priority, but the priority order of SD1 to SD5 is not particularly limited.

In addition, in the above embodiment, an example was described in which a performance image is combined with a pattern image and then output, but the image onto which the pattern image is combined is not limited to a performance image. For example, the pattern image may be combined with an image prepared in advance and then output.

In addition, in the above embodiment, the information processing device of the present invention is described as being separate from the electronic musical instrument, but the information processing device may be built into the electronic musical instrument.

In addition, for example, in the above embodiment, an example was disclosed in which a semiconductor memory or a hard disk was used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, portable recording media such as SSDs and CD-ROMs can be applied. Furthermore, carrier waves can also be applied as a medium for providing data for the program according to the present invention via a communication line.

In addition, the detailed configurations and operations of the above embodiments may be modified as appropriate without departing from the spirit of the invention.

Although the embodiments and modifications of the present invention have been described above, the technical scope of the present invention is not limited to the above-mentioned embodiments and modifications, but is defined based on the claims. Furthermore, the technical scope of the present invention also includes an equivalent range of modifications that are not related to the essence of the present invention from the claims.

100 Performance system 1 Information processing device 11 Control unit 12 Memory unit 121 Performance image generation application 13 Input unit 14 Display unit 15 Communication unit 16 Bus 2 Electronic musical instrument 21 Control unit 22 Memory unit 23 Keyboard 24 Input unit 25 Display unit 26 Communication unit 27 Output unit 28 Bus

Claims (14)

On the computer,
calculating an evaluation value for determining an impression given by the performance from performance information based on the performance;
a pattern image that visually conveys an impression of the performance based on the calculated evaluation value is synthesized with the image and output.
A program for executing a process. The program according to claim 1, wherein the image is a performance image based on features obtained by analyzing the performance information. The program according to claim 1, wherein the pattern image is a background image that forms the background of the image. The program according to claim 1, wherein the pattern image is a mask image for masking a portion of the image. The process comprises:
calculating an evaluation value for determining whether the performance gives one impression of a predetermined antonym pair, gives the other impression, or gives neither impression;
when it is determined that the performance gives the one of the impressions based on the evaluation value, a first pattern image that visually gives the one of the impressions is synthesized with the image, and outputted;
when it is determined that the performance gives the impression of the other person based on the evaluation value, a second pattern image that visually gives the impression of the other person is synthesized with the image and output.
The program according to claim 1. The program according to claim 2, wherein the feature is at least one of the velocity, tempo, key, pitch, and number of chords in the performance. The program according to claim 5, wherein the antonym pairs are at least one of delicate and powerful, slow and fast, sad and happy, heavy and light, and regular and free. The process comprises:
6. The program according to claim 5, further comprising: calculating the evaluation value for determining whether the performance gives a delicate impression, a powerful impression, or neither impression, based on an average velocity of the performance over a predetermined period of time. The process comprises:
6. The program according to claim 5, further comprising: calculating the evaluation value for determining whether the performance gives the impression of being slow, fast, or neither, based on an average tempo of the performance over a predetermined period of time. The process comprises:
6. The program according to claim 5, further comprising: calculating the evaluation value for determining whether the performance gives a sad impression, a happy impression, or neither impression, based on an average tonality and tempo of the performance over a specified period of time. The process comprises:
The program according to claim 5, further comprising: calculating an evaluation value for determining whether the performance gives a heavy impression, a light impression, or neither, based on average pitch, number of chords, and tempo over a specified period of the performance; and calculating the evaluation value. The process comprises:
6. The program according to claim 5, further comprising: calculating the evaluation value for determining whether the performance gives an impression of regularity, an impression of freedom, or neither, based on the variance or standard deviation of the tempo over a specified period of the performance. calculating an evaluation value for determining an impression given by the performance from performance information based on the performance;
a pattern image that visually conveys an impression of the performance based on the calculated evaluation value is synthesized with the image and output.
An information processing device having a control unit. The computer
calculating an evaluation value for determining an impression given by the performance from performance information based on the performance;
a pattern image that visually conveys an impression of the performance based on the calculated evaluation value is synthesized with the image and output.
Image generation method.

Images