JP7585681B2 - Performance information prediction device, performance model training device, performance information generation system, performance information prediction method, and performance model training method - Google Patents

Description

This disclosure relates to a performance information prediction device, a performance model training device, a performance information generation system, a performance information prediction method, and a performance model training method.

There is an electronic musical instrument called a guitar controller (or guitar synthesizer) that converts the string vibration waveform of an instrument such as a guitar into an electric signal using a magnetic or piezo pickup, analyzes the pitch and volume, and converts it into digital performance data such as MIDI (Musical Instrument Digital Interface) messages. Unlike dedicated guitar controllers that are only used to play sound sources, this type of controller retains the shape and functions of a normal guitar, but has the great advantage of being equipped with independent pickups for each string to obtain performance information, making it possible to play MIDI, and is said to be the most common form.

Japanese Patent Application Publication No. 9-6339 JP 2000-105590 A

However, one of the major problems with these instruments that has remained unsolved for many years is the so-called tracking error, where the performance information is converted into unintended performance information when the strings are plucked. This is because the pitch is detected by observing the peaks of the input signal waveform and the cycle of the zero crossing points, and the sound generation method is determined only from the amount of change in the input envelope, so it is fooled by the transient performance noise caused by picking or tapping when plucking the strings, and the complex harmonic movements of the strings.

For example, when it comes to picking noise, there are cases where the friction noise that occurs when the pick touches the string before plucking, and the picking noise caused by the vibration of the very short length of the string between the pick and the bridge are recognized as playing sounds. This can result in note information that is far higher in pitch than the playing pitch that corresponds to the fret position where the string is actually pressed.

In addition, when it comes to harmonics, there are cases where harmonics are not produced by an intentional playing technique, but rather the amount of harmonics contained in the string vibration during normal playing is so high that it is difficult to distinguish between the fundamental tone and the overtone, and the harmonic pitch is recognized as the played note. The most common case is that the second overtone is mistaken for a pitch one octave higher than the actual pitch, but it is also easy to mistake the third overtone for the fundamental tone.

Regarding the misrecognition of picking and legato playing, when the sound production technique of guitar strings is judged and performance information is added, there are cases where it is misunderstood as an unintended sound production technique. For example, the sound production techniques of guitar strings can be classified as follows based on the characteristics of the sound.
A. Normal plucking with picking or fingers. B. Hammer-on (using the fingers of the fret hand to strike or touch a higher fret on a string, changing the pitch) or tapping (using the fingers of the opposite hand to strike the string).
C. Pull-off technique: Pull the string slightly with the finger pressing the fret and then release it, or use the same finger as above to pull and release the string to pluck it.
D. A glissando or slide technique in which the pitch is changed by pressing and sliding a finger over the currently sounding string. This is not interpreted as a new sound being produced by plucking the string in a MIDI message, but is usually represented as a pitch change.

Of these, cases A, B, and C generally generate new sounding information, while case D is determined to be legato playing and generates pitch bend information for the currently sounding note.

It is believed that these playing styles can be determined not only by changes in the volume envelope when sound is produced, but also by analyzing changes in the levels of the harmonics of various noises that occur during transients, but this type of analysis has been difficult using conventional methods.

Furthermore, the frequency components and changes when plucking the strings vary greatly depending on the different characteristics of each guitar, the player's habits, the shape and material of the pick, and in fingerpicking, the hardness of the player's skin, so it is necessary to take individual characteristics into account when making a judgment. However, there is currently nothing that takes such factors into account when making a judgment, and in reality, there are almost no instruments that can judge playing style itself.

In view of the above problems, the objective of this disclosure is to provide technology for converting the performance of an electronic stringed instrument into performance information with high accuracy.

In order to solve the above-mentioned problems, one aspect of the present disclosure relates to a performance information prediction device including: a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance, and acquires, based on the spectral data frame, a spectral feature data frame composed of frequencies of a predetermined number of top peaks contained in the spectral data frame; and a performance information prediction unit that utilizes a trained performance model to predict performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times before and after the reference time, wherein the trained performance model acquires the spectral feature data frame at the reference time, a first number of spectral feature data frames before the reference time, and a second number of spectral feature data frames after the reference time, and outputs a playing style and note numbers of the stringed instrument .

According to this disclosure, it is possible to convert the performance of an electronic stringed instrument into performance information with high accuracy.

FIG. 1 is a schematic diagram illustrating a guitar controller according to one embodiment of the present disclosure. A diagram showing the configuration of performance information according to one embodiment of the present disclosure. FIG. 1 illustrates a tablature according to one embodiment of the present disclosure. FIG. 1 illustrates an external view of a guitar controller according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing a hardware configuration of a guitar according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing a hardware configuration of a control device according to an embodiment of the present disclosure. 1 is a schematic diagram illustrating an operation of a musical performance model training device according to an embodiment of the present disclosure. 1 is a block diagram showing a functional configuration of a musical performance model training device according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating a spectral data frame according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating a characterization data frame according to an embodiment of the present disclosure. A diagram showing the architecture of a performance model according to one embodiment of the present disclosure. A diagram showing the architecture of a performance model according to another embodiment of the present disclosure. 13 is a flowchart illustrating a performance model training process according to an embodiment of the present disclosure. 1 is a schematic diagram illustrating an operation of a performance information prediction device according to an embodiment of the present disclosure. 1 is a block diagram showing a functional configuration of a performance information prediction device according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating volume detection and pitch detection according to one embodiment of the present disclosure. 11 is a flowchart illustrating a performance information prediction process according to an embodiment of the present disclosure.

In the following embodiment, a guitar controller is disclosed that generates performance information (e.g., MIDI messages) from a string vibration waveform generated by playing a guitar. Note that the present disclosure is not limited to a guitar controller, and may be applied to any other performance information generating device that generates performance information from the performance of a stringed instrument having a string vibration waveform extraction function.
[Summary of the Disclosure]
1 , a guitar controller 10 according to an embodiment of the present disclosure includes a guitar 50 and a control device 100. The guitar controller 10 generates performance information from a string vibration waveform generated by playing the guitar 50, using a performance model realized as a machine learning model such as a neural network.

The performance information according to one embodiment of the present disclosure indicates the performance type of sound production information, mute information, and pitch change information, as shown in FIG. 2.

The sound generation information indicates the note number and playing style determined by the performance model as a classification model, and the strength of the string plucking determined by envelope detection. The playing styles are classified into seven types, for example: 0) picking with a pick, 1) finger picking, 2) hammering on (tapping), 3) pulling off, 4) mute picking, 5) open harmonics, and 6) picking harmonics. When sound generation is expressed by a MIDI message, the playing style may be represented by Control Change: 0xBn, 0x46, vv, and the note number and string plucking strength may be represented by Note On: 0x9n, kk, vv.

Also, the mute information is detected by envelope detection and indicates 0) sound stop and 1) replacement. When mute is expressed by a MIDI message, it may be expressed by Control Change: 0xBn, 0x46, vv and Note Off: 0x8n, kk, vv.

The pitch change information is detected by zero cross counting, and indicates, for example, half-tone choking up, half-tone choking down, whole-tone choking up, whole-tone choking down, one-tone and a half-tone choking up, one-tone and a half-tone choking down, two-tone choking up, two-tone choking down, and slide. When expressing pitch change by MIDI message, it may be expressed by Pitch Bend: 0xEn, ll, mm.

In the embodiment shown in FIG. 1, the guitar controller 10 has two operating modes: a training mode for training a performance model, and a performance mode for predicting performance information using the trained performance model, and the control device 100 has a performance model training device 200 used in the training mode, and a performance information prediction device 300 used in the performance mode.

First, in the training mode, the guitar controller 10 acquires training data from the training performance information database 80. The training data is composed of, for example, a pair of music score data (e.g., TAB scores, etc.) and a MIDI file corresponding to the music score data. The TAB scores may be written according to a well-known notation, such as that shown in FIG. 3. When the user plays the guitar 50 according to the music score of the acquired training music score data, the performance model training device 200 inputs string vibration information generated by the guitar 50 based on the user's performance to the performance model to be trained, compares the MIDI message as performance information output from the performance model with the training MIDI file, and trains the performance model so that the error between them is reduced. In the present disclosure, the string vibration waveform data is converted into spectrum data by a fast Fourier transform (FFT), and performance information is acquired from the performance model using spectral feature data characterized based on a predetermined number of peaks in the spectral data. When training is completed, the performance model training device 200 provides the trained performance model to the performance information prediction device 300.

Next, in the performance mode, when the user plays the guitar 50, the performance information prediction device 300 inputs string vibration information generated by the guitar 50 based on the user's performance into the trained performance model, and acquires performance information such as MIDI messages. The acquired performance information is transmitted to, for example, an external playback device or computer, and the user can play back the user's performance via the playback device or use the performance information on the computer.

This reduces tracking errors when converting the performance of an electronic string instrument into performance information, and makes it possible to determine playing style with high accuracy.

It should be noted that the guitar controller 10 according to the embodiment described below has the performance model training device 200, but the present disclosure is not limited to this. For example, the performance model may be trained by an external computer or server, and the trained performance model and/or update information of the performance model may be provided to the performance information prediction device 300 from the external computer or server.
[Hardware configuration]
Next, the physical configuration of the guitar controller 10 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the external appearance of the guitar controller 10 according to one embodiment of the present disclosure.

As shown in FIG. 4, the guitar controller 10 is a separate type performance information generation system consisting of an interconnected guitar 50 and a control device 100.

The guitar 50 is a normal electric guitar equipped with a hexa-divided pickup for picking up the independent vibrations of each of the six strings, a MIDI volume for controlling the volume of the performance information, and an up-down switch for switching the patch memory number up or down for the control device 100. This information and the output of the normal pickup are sent to the control device 100 via a multi-cable. Power is also supplied from the control device 100 via the multi-cable. The hexa-divided pickup in this embodiment is a magnetic pickup, the same as a normal pickup.

On the other hand, the control device 100 receives input of guitar string vibrations and generates performance information in MIDI format. The destination of the performance information may be, without limitation, a sound source unit, a computer, etc. As shown in FIG. 1, the control device 100 has a foot switch for switching the bank number and number of the patch memory in which various settings are stored, a CONTROL switch that can assign and send any performance message, and a foot pedal. The number of the currently selected patch memory is displayed on the BANK/NUM screen. The main display device is an LCD, but a touch panel is attached to the screen. A rotary encoder for inputting data is also provided on the panel. As terminals, the input terminal GUITAR INPUT for the multi-cable from the guitar 50, the audio output terminal GUITAR OUT for the normal pickup, the output terminal MIDI OUT for the MIDI performance signal, the connection terminal USB to HOST with the host computer, and the AC power input terminal AC POWER are provided.

Next, the hardware configuration of the guitar 50 according to one embodiment of the present disclosure will be described with reference to FIG. 5. FIG. 5 is a block diagram showing the hardware configuration of the guitar 50 according to one embodiment of the present disclosure.

As shown in FIG. 5, the guitar 50 sends the signal from the hexa-divided pickup buffer amplifier, the MIDI volume control, the patch memory up/down switch, and the normal pickup signal to the control device 100 via a multi-cable. The three normal pickups are selected by a pickup selector, and the signal is sent to the control device 100 after passing through the tone control circuit, the volume control circuit, and the buffer amplifier.

Next, the hardware configuration of the control device 100 according to one embodiment of the present disclosure will be described with reference to FIG. 6. FIG. 6 is a block diagram showing the hardware configuration of the control device according to one embodiment of the present disclosure.

As shown in FIG. 6, the control device 100 is composed of a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The CPU manages the overall functions and processing of the control device 100, and the DSP executes waveform analysis processing that requires high speed processing. The CPU bus is connected to the RAM used by the CPU, Flash ROM, an LCD controller that controls the LCD, an I/O interface connected to various I/O devices, the DSP, a USB interface, and a MIDI interface. In addition, the I/O interface is connected to a foot switch, a rotary encoder, an LCD touch panel, the MIDI volume of the guitar 50, an A/D converter for detecting the position of the pedal of the control device 100, and an LED for displaying the patch memory number. Although only one A/D converter is shown, the input source is switched in a time-division manner by a multiplexer to read values. The DSP, which is connected to a dedicated RAM and Flash ROM, is also connected to an independent A/D converter that quickly converts the output of the six strings of the hexa-divided pickup into a digital signal, allowing for high-speed analysis and processing.

However, the guitar 50 and the control device 100 are not limited to the above-mentioned hardware configuration, and may be realized by any other appropriate hardware configuration.
[Performance model training device]
Next, the musical performance model training device 200 according to an embodiment of the present disclosure will be described with reference to Figures 7 to 12. Figure 7 is a schematic diagram showing the operation of the musical performance model training device 200 according to an embodiment of the present disclosure.

As shown in FIG. 7, the performance model training device 200 trains the performance model using the training data stored in the training performance information database 80 that stores the training data. Specifically, the training data is composed of a pair of training score data and training performance information corresponding to the score data, and a score (e.g., TAB score, etc.) displayed based on the training score data is displayed to the performer, who plays the guitar 50 under tempo control by a metronome. The string vibration waveform data representing the performance is provided to the performance model training device 200, which converts the acquired string vibration waveform data into a spectral feature data frame, which will be described in detail below, and inputs the spectral feature data frames at the reference time and around the reference time to the performance model to be trained. The performance model training device 200 then compares the output from the performance model with the pronunciation information (e.g., note number and playing style, etc.) of the training performance information, and updates the parameters of the performance model according to the error. The performance model training device 200 repeats the above-described process until a predetermined termination condition is satisfied, and optimizes the performance model so that the output from the performance model approaches the pronunciation information of the training performance information.

Figure 8 is a block diagram showing the functional configuration of a performance model training device 200 according to one embodiment of the present disclosure.

As shown in FIG. 8, the performance model training device 200 has a preprocessing unit 210 and a performance model training unit 220.

The pre-processing unit 210 generates a spectral data frame from string vibration waveform data representing a stringed instrument performance performed according to the training performance information, and converts the spectral data frame into a spectral feature data frame.

Specifically, when the guitar 50 is played by a player, the guitar 50 acquires string vibration waveform data indicating time and the amplitude of each string, as shown in FIG. 9, and transmits the data to the performance model training device 200. That is, since the guitar 50 has six strings, six types of string vibration waveform data are generated. The pre-processing unit 210 performs a fast Fourier transform (FFT) on the string vibration waveform data of each string to acquire spectral data. Specifically, the pre-processing unit 210 may extract string vibration waveform frames with a window width w (e.g., W=512, 25.6 msec, etc.) that overlap on the time axis from the string vibration waveform data, and perform an FFT every I samplings (I=64, 3.2 msec, etc.) to convert each string vibration waveform frame into a spectral data frame.

After conversion to a spectral data frame, the preprocessing unit 210 characterizes the spectral data frame by a predetermined number of the top peaks of each spectral data frame. For example, when characterizing a spectral data frame by the top four peaks, the preprocessing unit 210 characterizes the spectral data frame by the frequencies of the top four peaks (maxima) on the frequency axis in the spectral data frame as shown in FIG. 10, and constructs a spectral characterization data frame by the frequencies of the four peaks. This characterization reduces the data size and emphasizes the peak heights and time changes in height from the peaks that are assumed to be related to the playing style and note number to be predicted, which is believed to improve the accuracy and speed of the performance model training process.

The preprocessing unit 210 generates spectral characterization data consisting of a predetermined number of peaks extracted in this manner and provides it to the performance model training unit 220.

The performance model training unit 220 uses the training performance information to train a performance model that predicts performance information of a stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time. Here, when predicting the playing style and note number at the reference time to be predicted, the performance model to be trained acquires as input not only the spectral feature data frame at the reference time but also the spectral feature data frames at times before and after the reference time, and outputs the playing style and note number at the reference time. For example, the performance model training unit 220 may input the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time to the performance model. Here, the predetermined numbers p and n may be the same or different predetermined values. For example, it is preferable that the predetermined numbers p and n are set to values such that the time lag between the player plucking the strings of the guitar 50 and the output of the performance information in the performance information prediction device 300 is not noticeable by the player.

By using a spectral feature data frame for a certain time range in this way, it is possible to determine whether a new pluck has occurred while taking into account the context between frames, and to determine the change in the pluck over time.

In addition, if there is no sound at the reference time, i.e., if the reference time is in a mute state, the performance model may be trained to output a value indicating that detection is not possible.

In one embodiment, the performance model may be realized by a neural network. For example, the performance model may be a neural network having a network architecture as shown in FIG. 11. In this case, the performance model training unit 220 inputs a spectral feature data frame of a reference time and (p+n) spectral feature data frames around the reference time to the input layer of the neural network, and obtains the rendition style number Var and the note number Note from the output layer through calculations in the intermediate layer.

In another embodiment, the performance model may be realized by a recurrent neural network as shown in Fig. 12. In this case, the performance model training unit 220 may use a spectral feature data frame having a wider time range than the time range of p and n described above. For example, the spectral feature data frame at the reference time t, b (b>p) spectral feature data frames immediately before the reference time, and f (f>n) spectral feature data frames immediately after the reference time may be input to the input layer Xt _-b , ..., Xt _-1 , _Xt , _Xt+1 , ..., Xt _+f of the recurrent neural network, and the rendition style number Var and the note number Note may be obtained from the output layer through calculations in the intermediate layer. The recurrent neural network is suitable for processing time series data, and is considered to be able to predict the rendition style number Var and the note number Note with high accuracy.

The performance model to be trained may be a pre-trained machine learning model, and the performance model training unit 220 may fine-tune the pre-trained performance model by the above-mentioned training process. This makes it possible to build a highly accurate performance model with less training data compared to training a performance model from an initial machine learning model.

When the performance model obtains the playing style and note number from the performance model, the performance model training unit 220 compares the obtained playing style and note number with the playing style and note number of the training performance information, and updates the parameters of the performance model so that they match. For example, if the performance model is realized by a neural network, the performance model training unit 220 may update the parameters of the neural network according to the comparison result according to the well-known backpropagation method.

The performance model training unit 220 repeats the above-mentioned process to train the performance model until a predetermined termination condition is satisfied, and when the predetermined termination condition is satisfied, passes the performance model at that time point as a trained performance model to the performance information prediction device 300. Here, the predetermined termination condition may be, for example, that all prepared training data has been processed.
[Performance model training process]
Next, a musical performance model training process according to an embodiment of the present disclosure will be described with reference to Fig. 13. The musical performance model training process is realized by the musical performance model training device 200 described above, and may be realized by, for example, a processor executing a program or an instruction. Fig. 13 is a flowchart showing the musical performance model training process according to an embodiment of the present disclosure.

As shown in FIG. 13, in step S101, the performance model training device 200 selects training performance information from the training performance information database 80. Specifically, the performance model training device 200 may automatically select training performance information randomly, sequentially, or by user selection.

In step S102, the performance model training device 200 converts the performance information into tablature display information.

In step S103, the performance model training device 200 displays the TAB score on the LCD or other display of the control device 100.

In step S104, the performance model training device 200 starts the MIDI player in sync with the tempo of the performance information.

In step S105, the performance model training device 200 starts the metronome in time with the tempo.

In step S106, the MIDI player plays the performance information.

In step S107, the metronome plays back the performance information. This completes the preparations for acquiring the performer's performance, and the performer begins playing.

In step S108, the performance model training device 200 initializes the string number s to 0. Since the guitar 50 has six strings, the string number s can take a value from 0 to 5.

In step S109, the performance model training device 200 stores the s-channel pronunciation information generated by the MIDI player in the pronunciation information memory p.

In step S110, the performance model training device 200 obtains the string vibration waveform of string number s, which represents the performance by the performer, from the buffer, generates a spectral feature data frame, and stores it in a ring buffer or the like.

In step S111, the performance model training device 200 inputs the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time to the performance model to be trained.

In step S112, the performance model training device 200 stores the performance model output result in memory o.

In step S113, the performance model training device 200 compares the sound production information (e.g., playing style number and note number) in memory p with the output result of the performance model in memory o.

In step S114, the performance model training device 200 determines whether there is a difference between the pronunciation information in memory p and the output result in memory o.

If there is a significant difference (S114: Yes), in step S115, the performance model training device 200 applies optimization information to the performance model to update the performance model from the difference, and proceeds to step S116. On the other hand, if there is no significant difference (S114: No), the performance model training device 200 proceeds to step S116 without updating the performance model.

In step S116, the performance model training device 200 increments the string number s by 1 to process the next string.

In step S117, the performance model training device 200 determines whether the performance model update process has been completed for all strings, and if the update process has not been completed for all strings (S117: Yes), it returns to step S109.

In step S118, the performance model training device 200 determines whether the entire performance information has been processed, and if the entire performance information has not been processed (S118: No), the process returns to step S106.

In step S119, the performance model training device 200 stops the metronome and the MIDI player.

In step S120, the performance model training device 200 determines whether an end operation has been performed by the user or the like. If no end operation has been performed (S120: No), the performance model training device 200 returns to step S101 and selects the next performance information. If an end operation has been performed (S120: Yes), the performance model training device 200 terminates the processing.
[Performance Information Prediction Device]
Next, the performance information prediction device 300 according to an embodiment of the present disclosure will be described with reference to Figures 14 to 16. Figure 14 is a schematic diagram showing the operation of the performance information prediction device 300 according to an embodiment of the present disclosure.

The performance information prediction device 300 uses the performance model trained by the performance model training device 200 to predict performance information (e.g., MIDI messages, etc.) from the performer's performance of the guitar 50. Specifically, as shown in FIG. 14, when the performance information prediction device 300 acquires string vibration waveform data representing a guitar performance from the guitar 50, it performs a fast Fourier transform on the acquired string vibration waveform data to generate a spectral data frame of a predetermined window width having overlapping portions on the time axis. The performance information prediction device 300 then identifies a predetermined number of top peaks in terms of frequency in each spectral data frame, and generates a spectral feature data frame by extracting the identified peaks. For example, these preprocessing steps may be implemented by a DSP.

To predict the performance information at the reference time, the performance information prediction device 300 inputs the spectral feature data frames of a certain time range before and after the reference time, i.e., the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time, into the trained performance model to obtain sound information including the playing style and note number. The performance information prediction device 300 also performs volume detection and pitch detection on the spectrum at the reference time to detect the volume and pitch of the performance. The performance information prediction device 300 generates silencing information and pitch change information based on the detected volume and pitch, respectively, and generates velocity information indicating the strength of the plucking based on the volume and adds it to the sound information. In this way, the performance information prediction device 300 generates performance information (e.g., MIDI messages, etc.) for each time including sound information, silencing information, and/or pitch change information from the spectral feature data frames at each time, and transmits it to an external device (e.g., a playback device, a computer, etc.). For example, these performance information generation processes may be realized by a CPU.

Figure 15 is a block diagram showing the functional configuration of a performance information prediction device 300 according to one embodiment of the present disclosure.

As shown in FIG. 15, the performance information prediction device 300 has a preprocessing unit 310 and a performance information prediction unit 320.

The preprocessing unit 310 generates a spectral data frame from the string vibration waveform data representing the performance of a stringed instrument, and converts the spectral data frame into a spectral feature data frame. As with the preprocessing unit 210, when the guitar 50 is played by the performer, the preprocessing unit 310 acquires string vibration waveform data of each string from the guitar 50, performs a fast Fourier transform (FFT) on the string vibration waveform data of each string, and acquires spectral data. Specifically, under the same settings as the preprocessing unit 210, the preprocessing unit 310 may extract string vibration waveform frames of a window width w that overlap with respect to the time axis from the string vibration waveform data, perform an FFT for each sampling, and convert each string vibration waveform frame into a spectral data frame. After conversion into a spectral data frame, the preprocessing unit 310 generates a spectral feature data frame based on a predetermined number of top peaks of each spectral data frame, and stores the spectral feature data frames in a certain time range around the reference time in a ring buffer or the like.

The performance information prediction unit 320 uses a trained performance model to predict performance information of a stringed instrument performance from the spectral feature data frame at the reference time and the spectral feature data frames before and after the spectral feature data frame at the reference time. Specifically, the performance information prediction unit 320 inputs the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time into the trained performance model, and obtains the playing style and note number at the reference time.

In parallel with this, the performance information prediction unit 320 performs volume detection and pitch detection on the spectral feature data frame at the reference time. For example, for volume detection, the performance information prediction unit 320 may calculate the sum of the frequency levels of a predetermined number of peaks in the spectral feature data frame as shown in FIG. 16(a), and determine the calculated total frequency level as the volume at the reference time. If the trained performance model does not detect pronunciation at the reference time, or if the detected volume is equal to or less than a predetermined threshold that is recognized as a muted state, the performance information prediction unit 320 determines that there was no pronunciation and outputs muted information as the performance information. Otherwise, the performance information prediction unit 320 determines that there was pronunciation, sets the detected volume as the velocity value of the pronunciation, and includes the velocity value in the pronunciation information together with the playing style and note number output from the performance model.

For pitch detection, the performance information prediction unit 320 determines the minimum frequency level among a predetermined number of peaks in the spectrum feature data frame as the pitch of the plucked string, as shown in FIG. 16(b), and generates pitch information. If there is a difference from the most recent pitch information or pronunciation information, the performance information prediction unit 320 determines that a pitch change has occurred and outputs pitch change information.

The pre-processing unit 310 and the performance information prediction unit 320 execute the above-mentioned processes in parallel for all the strings.
[Performance Information Prediction Processing]
Next, a performance information prediction process according to an embodiment of the present disclosure will be described with reference to Fig. 17. The performance information prediction process is realized by the above-mentioned performance information prediction device 300, and may be realized by, for example, a processor executing a program or an instruction. Fig. 17 is a flowchart showing the performance information prediction process according to an embodiment of the present disclosure.

As shown in FIG. 17, in step S201, the performance information prediction device 300 initializes the string number s to 0. Since the guitar 50 is made up of six strings, the string number s can take a value between 0 and 5.

In step S202, the performance information prediction device 300 generates a spectral feature data frame from the string vibration waveform data and performs volume detection on the spectral feature data frame at the reference time.

In step S203, the performance information prediction device 300 determines whether the detected volume I is less than a predetermined threshold. If the volume I is equal to or greater than the predetermined threshold (S203: No), the performance information prediction device 300 determines that sound is being produced and proceeds to step S206.

On the other hand, if the volume I is less than the predetermined threshold (S203: Yes), the performance information prediction device 300 determines in step S204 whether the sound is being produced at the reference time. For example, this determination may be made based on whether sound production information was previously output from the trained performance model. If the sound is being produced (S204: Yes), the performance information prediction device 300 generates mute information corresponding to the note number output from the performance model in step S205. If the sound is not being produced (S204: No), the performance information prediction device 300 proceeds to step S206.

In step S206, the performance information prediction device 300 extracts from the buffer the spectral feature data frame for the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time.

In step S207, the performance information prediction device 300 inputs the extracted spectral feature data frame into the performance model.

In step S208, the performance information prediction device 300 determines whether or not sound information including the playing style and note number has been output from the performance model. If sound information has been output (S208: Yes), the performance information prediction device 300 assigns the output playing style and note number to variables v and k, respectively, in step S209. On the other hand, if sound information has not been output (S208: No), the performance information prediction device 300 proceeds to step S215.

In step S210, the performance information prediction device 300 determines whether a sound has been produced. If a sound has been produced (S210: Yes), the performance information prediction device 300 generates mute information for note number K0 of the previous sound production event in step S211. On the other hand, if no sound has been produced (S210: No), the performance information prediction device 300 proceeds to step S212.

In step S212, the performance information prediction device 300 generates pronunciation information including the velocity converted from the playing style number v, the note number k, and the volume I.

In step S213, the performance information prediction device 300 assigns k to the note number K0 of the previous sound production event.

In step S214, the performance information prediction device 300 identifies the pitch that corresponds to the previously generated pitch P0=k.

In step S215, the performance information prediction device 300 performs pitch detection on the spectral feature data frame at the reference time and stores the detected pitch in p.

In step S216, the performance information prediction device 300 determines whether p = P0. If p = P0 (S216: Yes), the performance information prediction device 300 proceeds to step S218, and if p = P0 is not true (S216: No), in step S217, the performance information prediction device 300 generates pitch bend information based on the difference from p.

In step S218, the performance information prediction device 300 increments the string number s by 1 to perform the above-described process on the next string.

In step S219, the performance information prediction device 300 determines whether all strings have been processed. If all strings have been processed (S219: No), the performance information prediction process ends. If not (S219: Yes), the above-described process is repeated for the next string.

In the above-described embodiment, a performance information prediction system is described in which a performance model that predicts performance information from string vibration waveform data of a stringed instrument such as a guitar 50 is trained, and the performance information is predicted using the trained performance model. However, the present disclosure is not limited to this, and may be applied to a wind instrument. In other words, the present disclosure may be applied to a performance information prediction system in which a performance model that predicts performance information from air vibration waveform data of a wind instrument is trained, and the performance information is predicted using the trained performance model.

Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

The invention as originally claimed in the present application is set forth below.
[Additional Notes]
In one aspect of the present disclosure,
a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance, and obtains a spectral characterization data frame based on the spectral data frame;
a performance information prediction unit that predicts performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time by using a trained performance model;
The present invention provides a performance information prediction device having the above structure.

In one embodiment, the performance information may consist of sound generation information, mute information, and pitch change information.

In one embodiment, the sound information may include playing style and note number.

In one embodiment, the trained performance model may output the playing style and the note number.

In one embodiment, the trained performance model may obtain a spectral feature data frame at the reference time, a first number of spectral feature data frames before the spectral feature data frame at the reference time, and a second number of spectral feature data frames after the spectral feature data frame at the reference time, and output the playing style and the note number.

In one embodiment, the trained performance model may be realized by a neural network.

In one embodiment, the spectral characterization data frame may consist of a predetermined number of top peaks contained in the spectral data frame.

In one embodiment, the performance information may be described according to the MIDI protocol.

In another aspect of the present disclosure,
a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance performed in accordance with training performance information, and obtains a spectral characterization data frame based on the spectral data frame;
a performance model training unit that uses the training performance information to train a performance model that predicts performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time;
A performance model training device having the above structure is provided.

In another aspect of the present disclosure,
Electronic string instruments,
The performance information prediction device described above;
The above-mentioned performance model training device;
A performance information generating system having the above structure is provided.

In another aspect of the present disclosure,
generating, by one or more processors, a spectral data frame from string vibration waveform data representative of a stringed instrument performance, and obtaining a spectral characterization data frame based on the spectral data frame;
predicting performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time using a trained performance model;
A performance information prediction method is provided, which has the following steps:

In another aspect of the present disclosure,
generating, by one or more processors, a spectral data frame from string vibration waveform data representative of a stringed instrument performance performed according to training performance information, and obtaining a spectral characterization data frame based on the spectral data frame;
training a performance model using the training performance information, the one or more processors, to predict performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time;
A performance model training method is provided, which has the following:

10 Guitar controller 50 Guitar 100 Control device 200 Performance model training device 210 Pre-processing unit 220 Performance model training unit 300 Performance information prediction device 310 Pre-processing unit 320 Performance information prediction unit

Claims (9)

a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance, and obtains a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being configured from the frequencies of a predetermined number of top peaks included in the spectral data frame ;
a performance information prediction unit that predicts performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times before and after the reference time by using a trained performance model;
having
The trained performance model obtains the spectral feature data frame at the reference time, a first number of spectral feature data frames before the reference time, and a second number of spectral feature data frames after the reference time, and outputs a playing style and note number of a stringed instrument . The performance information prediction device according to claim 1, wherein the performance information is composed of sound generation information, sound suppression information, and pitch change information. 3. The performance information prediction device according to claim 2, wherein the sound generation information includes the playing style and the note number. 4. The apparatus according to claim 1, wherein the trained performance model is realized by a neural network. 5. The performance information prediction device according to claim 1, wherein the performance information is described in accordance with a MIDI protocol. a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance performed in accordance with training performance information, and obtains a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being configured from frequencies of a predetermined number of upper peaks included in the spectral data frame ;
a performance model training unit that uses the training performance information to train a performance model that predicts performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times before and after the reference time;
A performance model training device having the above structure. Electronic string instruments,
The performance information prediction device according to any one of claims 1 to 5 ,
A musical performance model training device according to claim 6 ,
A performance information generating system having the above configuration. one or more processors generating a spectral data frame from string vibration waveform data representing a stringed instrument performance, and obtaining a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being composed of frequencies of a predetermined number of top peaks included in the spectral data frame ;
predicting performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times around the reference time using a trained performance model;
having
The trained performance model obtains the spectral feature data frame at the reference time, a first number of spectral feature data frames before the reference time, and a second number of spectral feature data frames after the reference time, and outputs a playing style and note number of a stringed instrument . one or more processors generating a spectral data frame from string vibration waveform data representing a stringed instrument performance performed according to training performance information, and obtaining a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being composed of frequencies of a predetermined number of top peaks included in the spectral data frame ;
training a performance model using the training performance information, the one or more processors, to predict performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times around the reference time;
A performance model training method having the above structure.

Images