CN101093661B - A pitch tracking and playing method and system thereof - Google Patents

Abstract

A pitch tracking and playing method and system comprises a voice input processing module for receiving input voice and sampling and framing the input voice, a pitch and energy detection module for detecting pitch and energy of each voice frame, a note segmentation module for completing note segmentation according to energy detection results, a music score conversion module for converting the segmented notes into a music score, and a voice synthesis module for synthesizing the converted music score into a musical instrument digital interface file and playing the musical instrument digital interface file. Thus, human humming can be directly used as input, and the output is MIDI music. Furthermore, the invention can be applied to an embedded system by adopting measures of filtering by adopting a two-order low-pass filter during the preprocessing of pitch detection, calculating a cross-correlation function at intervals, reducing the search range of fundamental tone frequency and the like.

Description

A pitch tracking and playing method and system thereof

technical field

The present invention relates to a pitch tracking (Pitch Tracking) technology, in particular to a method for note segmentation and quantization in a pitch tracking and playback system. the

Background technique

Good music can not only cultivate elegant aesthetic taste, but also a very positive way to relieve stress, which plays an important role in modern society. With the development of digital sound processing technology, music creation and preservation methods are also constantly evolving. You can usually hear people humming unconsciously. In fact, humming is the most natural and direct way to make music or music inquiry, which requires adapting the humming signal into a music score and playing it back through a simple device. . In the general humming system, including humming recognition and arrangement, the main force has been focused on improving the accuracy of humming recognition in the past period of time. the

Humming recognition, that is, capturing pitch information and rhythm changes, is inseparable from pitch tracking technology. Pitch, also known as pitch or pitch frequency, characterizes the frequency at which the vocal cords vibrate during voiced sounds. Voiced sound refers to the sound produced by the quasi-periodic vibration of the vocal cords, and the corresponding pitch can be obtained; while unvoiced sound refers to the sound produced by the vocal cords without vibration, and there is no pitch parameter. the

The existing pitch tracking technology is implemented in the time domain and frequency domain by using various methods such as statistical and non-statistical signal processing, but there are still the following disadvantages:

1) The results of the pitch tracking method are mostly expressed in the form of graphs, texts, etc., and there is no suitable device to convert the tracking results into sound playback;

2) The notes (Note, such as "1, 2, 3, 4, 5, 6, 7--do, re, mi, fa, so, la, xi" in the numbered musical notation) of the pitch tracking device are not accurate enough , pitch recognition is not accurate enough;

3) The pitch tracking algorithm is too complex, takes up a lot of system memory, and often requires a large humming database, which is not suitable for embedded systems. the

Chinese patent application CN200410049328 published on April 20, 2005 discloses a humming arrangement system and method thereof. It mainly refers to compiling the input humming signal into a standard score and presenting it again. However, this patent has the following disadvantages: 1) the humming signal is only written as a standard score, and the sound cannot be played; 2) the note segmentation of the pitch tracking device is not accurate enough; 3) the note segmentation and recognition algorithm is a statistical Markov model, and requires matching of database parameters, the algorithm is relatively complex, takes up high resources, and cannot be applied to embedded systems, which limits the application of the entire system. the

U.S. Patent No. 5,986,199 published on November 16, 1999 discloses a music data sound input device. A method of embodiment is: preset notes are identified and selected from sound input signals, and auxiliary note information (including notes and Parameters such as note duration) are also extracted from the sound input signal, and auxiliary note information is used to generate synthesis engine parameters that modify preset notes to provide synthesized note output. Another embodiment is that the feature vector of note segmentation is used to select a preset file, which refers to a special instrument file from the instrument preset archive. According to the note split, select the preset note from the instrument preset file, and generate the note output corresponding to the specified instrument or instrument group. However, this patent uses a simple peak detection algorithm, and the note segmentation is not accurate. the

Contents of the invention

The technical problem to be solved by the present invention is to provide a pitch tracking and playback method and system, which can directly convert the tracking result into sound playback, and is suitable for use in embedded systems. the

In order to solve the problems of the technologies described above, the present invention provides a pitch tracking and playing system, comprising: a speech input processing module for receiving input sound and performing sampling and frame processing to it, for detecting the pitch of each speech frame A pitch and energy detection module for pitch and energy, a note segmentation device for completing note segmentation according to the energy detection result, and a music score conversion module for converting the note that completes segmentation into a score, characterized in that the The system also includes a speech synthesis module, which is used to synthesize the converted music score into a digital instrument interface file and play it. the

Further, the above-mentioned system can also have the following features: the pitch and energy detection module further includes a preprocessing unit, a normalized cross-correlation function calculation unit, a postprocessing unit and a pitch frequency search unit, wherein the preprocessing unit adopts A two-order low-pass filter is used for filtering. the

Further, the above-mentioned system may also have the following feature: the calculation of the cross-correlation function by the cross-correlation function calculation unit is performed at intervals. the

Further, the above-mentioned system may also have the following characteristics: the pitch frequency search unit adopts interval search when searching, and its search range is within 20-120. the

Further, the above-mentioned system can also have the following characteristics: the note segmentation device includes a peak detection module, a main control module, a note segmentation module, a storage module and a double peak determination module, wherein:

The peak detection module is used to count the voice frames whose energy involved in the energy continuous rising segment and the subsequent energy continuous falling segment from the initial voice frame of the voiced segment or the last voice frame of the previous energy falling segment is greater than the critical value number, if the number is greater than the third threshold, then it is determined that a section of energy curve corresponding to these speech frames constitutes a peak, the maximum energy value on this section of curve is the peak value of this wave peak, and the energy of the last speech frame on this section of curve The value is the valley value of the peak, and the peak and valley positions of the peak are the speech frames corresponding to the peak and the valley;

The storage module is used to save the parameters of the peak and the starting and ending positions of the voiced segment;

The double peak determination module is used to determine whether the ratio of the difference between the peak value of the second peak and the valley value of the first peak and the difference between the peak value of the first peak and the valley value is greater than the first threshold, and whether the peak value of the first peak and the second peak Whether the number of speech frames between the positions is greater than the second threshold, if they are all, the returned judgment result is success, otherwise the judgment result of failure is returned;

The main control module is used to perform note segmentation on each voiced segment composed of continuous speech frames with energy greater than a critical value, and further includes a first control unit, a second control unit, a third control unit, a fourth control unit and a fifth control unit. control unit, where:

The first control unit is used to call the peak detection module from the starting position of the voiced sound segment, and if no peak is detected, then end the processing of the voiced sound segment, otherwise the first peak detected is the first peak, and the The starting position of the voiced sound segment and the peak value, valley value, peak position and valley position of the first wave peak are stored in the storage module, triggering the second control unit to continue processing;

The second control unit is used to call the peak detection module. If the next peak is detected before the speech frame whose energy is less than the critical value is detected, the starting position of the voiced segment is output to the note segmentation device, and the third control unit is triggered to continue. Processing, otherwise, the voiced segment start and end positions are output to the note segmentation device;

The third control unit uses the detected next peak as the second peak, records its peak value, valley value, peak position and valley position, calls the double peak determination module, and if the returned determination result is successful, triggers the fourth control unit Continue processing, otherwise trigger the fifth control unit to continue processing;

The fourth control unit is used to call the peak detection module, such as detecting the next peak before detecting the speech frame whose energy is less than the critical value, outputting the position of the first peak and valley to the note segmentation device, and using the parameter of the second peak Overwrite the corresponding parameters of the saved first peak, and trigger the third control unit to continue processing; otherwise, output the first peak-valley position and the end position of the voiced segment to the note segmentation device;

The fifth control unit is used to invoke the peak detection module, such as detecting the next peak before detecting the speech frame whose energy is less than the critical value, and replacing the peak of the first peak with the larger value of the first peak and the second peak , replace the valley value of the first peak with a valley after the large peak or the minimum of the two valleys, and the peak position and valley position of the first peak are updated to the voice corresponding to the new peak and valley frame, then trigger the third control unit to continue processing; otherwise, output the end position of the voiced segment to the note segmentation device;

The note segmentation device is used to complete the note segmentation of the voiced segment by using two adjacent positions output during the processing of the voiced segment as the start and end positions of a note. the

Correspondingly, the pitch tracking provided by the present invention and playing method comprise the following steps:

(a) Sampling and framing the input sound;

(b) detect the pitch and energy of each speech frame;

(c) Complete the note segmentation according to the energy detection result, and then convert the note segmentation result into a score according to the detected pitch;

(d) Synthesizing the converted score into a digital instrument interface file and playing it. the

Further, the above method can also have the following characteristics: when the step (b) detects the pitch of the speech frame, it is further divided into the following steps:

(b1) Preprocessing the voice signal after frame processing, including removing the mean and low-pass filtering, and using a two-order low-pass filter for filtering;

(b2) compare the voice energy after the low-pass filter with a threshold, if it is higher than the threshold, it is judged as voiced sound, go to step (b3), otherwise, execute step 270;

(b3) Calculate the normalized cross-correlation function value ρ(t) of the voiced sound signal delayed by each frame;

(b4) Perform post-processing to obtain the most relevant ρ(t) and the corresponding optimal delay T;

(b5) Pitch frequency search, that is, whether the sample point t of the time delay corresponding to the maximum ρ (t) of the detection correlation is within the search range, if within the range, perform step (b6), otherwise, perform step (b7 );

(b6) think that this frame of speech is voiced sound, output the best delay T, and then according to the relationship between time and frequency, the pitch value corresponding to this frame of speech can be converted from this T, and end;

(b7) Think that the frame of speech is unvoiced, set the pitch to 0, and end. the

Further, the above method may also have the following feature: the calculation of the cross-correlation function value in the step (b3) is performed at intervals. the

Furthermore, the above-mentioned method may also have the following characteristics: the step (b5) adopts point-by-point search when performing the pitch frequency search, and the search range is within 20-120. the

Further, the above-mentioned method can also have the following characteristics: when the step (c) carries out note segmentation, for each voiced segment formed by continuous speech frames with energy greater than the critical value, perform the following steps:

(c1) take the first speech frame as the starting position of the voiced sound segment, as a wave peak is detected, the first wave peak detected is used as the first wave peak, record the voiced sound segment starting position and the peak value of the first wave peak, Valley, peak position and valley position, execute the next step, if no peak is detected, end;

(c2) continue to detect, if detect next wave peak before detecting the speech frame that energy is less than critical value, then output voiced sound segment initial position, execute step (c3), otherwise output this voiced sound segment initial position and end position, carry out step (c6);

(c3) Use the detected next peak as the second peak, record its peak value, valley value, peak position and valley position, and judge whether the difference between the peak value of the second peak and the valley value of the first peak and the peak value of the first peak The ratio of the difference between valleys is greater than the first threshold, and the number of speech frames between the peak positions of the first peak and the second peak is greater than the second threshold, if so, step (c4) is performed, otherwise step (c5) is performed;

(c4) Continue to detect, if the next peak is detected before the speech frame whose energy is less than the critical value is detected, the position of the first peak valley value is output, and the parameters of the second peak are used to overwrite the corresponding parameters of the saved first peak, and return Step (c3), otherwise, output the first peak and valley position and the end position of the voiced sound segment, and perform step (c6);

(c5) continue to detect, as detecting the next peak before detecting the speech frame with energy less than the critical value, cover the peak value of the first peak with the large value of the first peak value and the second peak value, and use the large value A valley after the peak or the small value of the two valleys overwrites the valley of the first peak, the peak position and valley position of the first peak are updated to the voice frame corresponding to the new peak and valley, and then return to the step (c3), otherwise, output the end position of the voiced sound segment, and perform step (c6);

(c6) Take the two output adjacent positions as the start and end positions of a note, and complete the note segmentation of the voiced segment. the

Wherein, when described step (a)～(e) carries out wave peak detection, is to compare the energy size of adjacent two speech frames one by one, counts from the initial speech frame of the voiced speech segment or the last speech frame of the preceding energy decline segment. The number of speech frames whose energy is greater than the critical value involved in the energy continuous rising section and the subsequent energy continuous falling section, if the number is greater than the third threshold, then it is determined that a section of the energy curve corresponding to these speech frames constitutes a peak, and this section The maximum energy value on the curve is the peak value of the peak, the energy value of the last speech frame on the curve is the valley value of the peak, and the peak and valley positions of the peak are the speech frames corresponding to the peak value and the valley value , the initial speech frame of the voiced segment or the last speech frame of the previous energy-decreasing segment is the initial position of the peak. the

By adopting the method and system of the present invention, human voice humming can be directly used as input, and the output is MIDI music, so that the change of pitch can be experienced without performing other conversion processing. The invention also simplifies the pitch detection part, so that the system memory occupies a small amount and can be applied to an embedded system. In addition, in the process of note segmentation, anti-interference, focused and layered double peak detection can be performed, and more effective detection parameters can be introduced to improve the accuracy of note segmentation. the

Description of drawings

Fig. 1 is a structural block diagram of a pitch tracking and playing system according to an embodiment of the present invention. the

Fig. 2 is an overall flow chart of the pitch tracking and playing method of the embodiment of the present invention. the

Fig. 3 is a schematic diagram of a pitch curve as an example. the

FIG. 4 is a flowchart of a pitch detection algorithm using a normalized cross-correlation function according to an embodiment of the present invention. the

Fig. 5 is a graph of energy and pitch used for splitting notes and converting them into scores according to an embodiment of the present invention. the

Fig. 6 is a flowchart of note segmentation according to an embodiment of the present invention. the

Fig. 7 is a structural block diagram of a musical note segmentation device according to an embodiment of the present invention. the

Detailed ways

The invention is applied to PCs and embedded systems, and is mainly used for tracking human voice humming, such as humming with "la", and can also be used for some electronic musical instruments. The input of the system is human voice, and the output is Midi music. Midi is the abbreviation of Musical-instrument-digital-interface (Musical-instrument-digital-interface), which is a file format for recording music scores. the

As shown in Figure 1, the pitch tracking and playing system of the present embodiment includes the following modules:

The speech input processing module is used to receive the input sound, sample and frame it, and output it to the pitch and energy detection module. the

The pitch and energy detection module is used to detect the pitch and energy of each speech frame, obtain the pitch curve and energy curve of the input sound, and then output it to the note segmentation device. the

The note segmentation device is used to complete the note segmentation according to the energy detection result and output it to the score conversion module. the

The musical score conversion module is used to convert the segmented notes into musical scores and output them to the speech synthesis module. the

Speech synthesis module, used for synthesizing converted scores into MIDI files and playing them. the

As shown in Figure 2, the overall process of the pitch tracking and playing method of the present embodiment comprises the following steps:

Step 10, sampling and framing the input sound;

The microphone of the hardware system can be used to collect vocal humming at a sampling rate of 8KHz and 16bit as input, or the sampling rate can be increased to 16KHz. Usually, the analysis and processing of speech is based on the short-term nature, so the input sound needs to be processed in frames. In this embodiment, when the input sound (8KHz sampling rate, 16bits representation) is divided into frames, each frame is 20ms. the

Step 20, pitch and energy detection;

Pitch is also called pitch frequency, and pitch detection and estimation is a very important problem in the field of speech processing. At present, there are many very mature algorithms for pitch detection, mainly divided into time domain, frequency domain and other popular methods. the

The characteristics of the speech signal are time-varying, and only remain relatively stable in a short time interval. Therefore, it is necessary to detect the pitch and energy of each speech frame to obtain the pitch curve and energy curve of the input sound. the

Step 30, note segmentation and conversion into score

After pitch detection, a pitch curve is obtained. Figure 3 is an example, which shows the change of pitch over time. Some notes are represented separately, and some notes need to be segmented together. Note segmentation requires a combination of pitch and energy. The test results are ready for completion. the

Although the pitch curve looks very intuitive, it is still not enough to make people feel personally as a parameter representing rich note changes. The system of this embodiment can not only detect the pitch but also play the processing results in real time, making people Have a full range of feeling to your notes, thus broaden its field of application. For example: it can be used as a comparative study of tonal languages, or retrieve songs by humming, or smart electronic toys, etc., which requires converting the note segmentation results into musical scores for playback. the

Step 40, using common synthesis technology, synthesize the converted musical score into a MIDI file and play it. the

The sound of electronic musical instruments can also be processed in the same way as above. the

In the above-mentioned step 20, the pitch detection algorithm of the normalized cross-correlation function (NCCF) of the present embodiment is basically the same as the existing method, and the difference will be introduced in the following steps, please refer to Fig. 4, the pitch detection algorithm includes The following steps:

Step 210, preprocessing the voice signal after the frame processing, including removing the mean value and 1000Hz low-pass filter processing;

Step 220, judging unvoiced and voiced sound frame by frame: comparing the voice energy after the low-pass filter with a threshold, if it is higher than the threshold, it is judged as voiced sound, go to step 230, otherwise, execute step 270;

Step 230, calculate the normalized cross-correlation function:

The normalized cross-correlation function value ρ(t) of the voiced signal delayed by each frame is calculated by the following formula:

$\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}{\sqrt{\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ]</span>$

Among them, s(n) represents the voice signal, N is the signal frame length, and one frame of voice is 20ms, after 8kHz sampling, then N=160, t is the sample point of time delay, and t is in the range of [0, N-1] It is divided into three areas, and the ρ(t) value is calculated at intervals in each area, and the degree of similarity (correlation) between the original signal and its delayed signal is compared to obtain the most similar ρ(t). A total of three ρ(t) values, corresponding time-delay samples and corresponding delay times are obtained for the three regions. the

Step 240, post-processing: compare the three ρ(t) values to obtain the most relevant ρ(t) and the corresponding optimal delay T;

Step 250, pitch frequency search: detect whether the sample point t of the time delay corresponding to the most correlated ρ(t) is within the range of [20, 120], if within this range, perform step 260, otherwise, perform step 270 ;

Step 260, consider the frame of speech as voiced sound, output the optimal delay T, and then convert the pitch value corresponding to the frame of speech from T according to the relationship between time and frequency. the

In step 270, consider the frame of speech as unvoiced, set the pitch to 0, and end. the

Correspondingly, the pitch and energy detection module can be divided into a pre-processing unit, a normalized cross-correlation function calculation unit, a post-processing unit, a pitch frequency search unit, and a pitch output unit. Compared with existing units, it has the following characteristics :

The existing preprocessing unit has adopted a 5th-order elliptic low-pass filter and a numerical filter. In this embodiment, the original 5th-order elliptic low-pass filter is changed to a Haar wavelet-based two-order low-pass filter, and the cutoff frequency Set to 1000Hz, and remove the numerical filter, thus greatly reducing the computational complexity. the

The calculation of the cross-correlation function is very complicated. The existing cross-correlation function calculation unit usually calculates the correlation function once for each point, but the unit of this embodiment performs the correlation calculation at every point, which reduces the calculation amount by half. the

The search range of the existing pitch frequency search unit is 20-147, and this unit of the present embodiment then narrows the search range to 20-120 (also can be a bit smaller) and takes every other point to search, has reduced the high-frequency pitch detection Error and reduction operations. the

The pitch detection part is the part with the greatest complexity. In this embodiment, through the improvement of the above units, the complexity is reduced without affecting the detection accuracy, so that the present invention can be applied to embedded systems. the

The note segmentation in the above step 30 is an important point of the present invention, which will be described in detail below. the

A, the purpose of note segmentation

For the pitch curve, it needs to be converted into notes (such as numbered notation "1, 2, 3, 4..."), that is, the score can be played, and there are only a few cases where a pitch corresponds to a note, and most cases are A pitch corresponds to several notes, which requires note segmentation to complete the score. the

B, the background of note segmentation

When people use lyrics to sing a cappella, the pitch curve becomes more complicated. On the one hand, it is affected by pitch changes, and on the other hand, the change of timbre (or transphonation) will also affect the ups and downs of the pitch curve, which strengthens the syncopation Difficulty of notes. For example, a piece of music itself has a change in pitch. The effect of playing the same pitch on a piano and a guitar is not the same. This is the effect of different timbres. It can be sung by singing lyrics. The latter lyrics are changing, which is more complicated than the former's pitch curve changes. The present embodiment method is mainly used for finishing the song that the former " la ", " clatter " sings. the

C, the existing note segmentation method

The purpose of note segmentation is to find each note, and because each note has high energy in the initial stage, then the energy begins to decrease and maintain until it disappears or is masked by the energy of the next note. This is why the system chooses the peak detection algorithm. The reason is shown in Figure 4. the

Peak detection is used to detect the peak position with higher energy in the signal. The existing technologies are:

A) Wavelet transform algorithm. Wavelet transform is the inheritance and development of traditional Fourier transform. It is mainly used in many fields such as signal processing, image processing, and voice processing, but the implementation is relatively complicated. the

B) Simple peak detection, or amplitude envelope detection, determines a peak through the first and second derivatives. Although this method is simple, its anti-interference performance is poor. the

C) Peak values are detected by parameters such as peak-to-peak value, half-peak value, and peak-to-valley value, which have been applied to the note segmentation technology. Although the obtained results are relatively stable, for the multi-peak in the voice signal (around a peak in the signal) There are small peaks) detection is not accurate. the

D) the musical note segmentation method of the present embodiment

This embodiment improves on the basis of the above-mentioned peak detection C), mainly in the following three aspects:

1) Pretreatment

Firstly, the energy values of each detected speech frame are sequentially input into a first-order low-pass filter for filtering, and the glitches in the energy curve are filtered out, so as to improve the anti-interference and detection effect;

2) Double peak detection

Please refer to Figure 5. There are speech frames with energy greater than the critical value and speech frames with energy less than the critical value on the entire energy curve. When the notes are segmented, only the continuous speech frames with energy greater than the critical value are processed, which are called voiced segments in the text. , the first and last speech frames in the continuous speech frames are the start position and end position of the voiced speech segment. On the energy curve, energy can be characterized by amplitude or power. Preferably, the critical value of energy can be set at 26dB-30dB, but the present invention is not limited thereto. the

The double peak detection method of this embodiment will be described below by taking a voiced sound segment as an example. As shown in Figure 6, the process includes the following steps:

Step 300, from the starting position of the voiced sound segment, that is, the first speech frame, judge whether the first peak is detected in the voiced sound segment, if yes, perform step 310, otherwise, directly end;

All need to carry out peak detection in the process of whole note segmentation, so first introduce the peak detection method that present embodiment adopts: compare the energy magnitude relation of the previous speech frame and the following speech frame in the speech frame sequence one by one, statistics from voiced sound The number of voice frames involved in the energy continuous rising segment and the subsequent energy continuous falling segment of the initial speech frame of the segment or the last speech frame of the previous energy falling segment, if the number is greater than the set threshold (the threshold is lower than Preferably it is 5-9, and this embodiment is 7), then it is considered that the energy curves corresponding to these speech frames form a peak. The maximum energy value on the energy curve of this section is the peak value of the peak, the energy value of the last voice frame on the energy curve of this section is the valley value of the wave peak, and the positions of the peak value and the valley value are respectively the voice frames corresponding to the peak value and the valley value, The starting position of the peak is the starting speech frame of the voiced speech segment or the last speech frame of the previous energy-decreasing segment. the

If the number of speech frames involved in the energy continuous rising segment and the subsequent energy continuous falling segment is less than or equal to the threshold, the energy curve corresponding to these speech frames is considered to be a small spike and is not processed. the

Step 310, using the detected first peak as the first peak, recording the voiced segment starting position and the relevant parameters of the first peak, including: the peak, valley, peak position and valley position of the peak;

Step 320, continue to detect, judge whether to detect next wave peak before detecting the speech frame of energy less than critical value, if not, represent that there is no wave peak before the end of this voiced speech segment, execute step 330, if detect next wave peak, Output this voiced sound segment starting position (step 320a), perform step 340;

Step 330, determine that the voiced sound segment corresponds to a note, output the starting position and the end position of the voiced sound segment, and end;

Step 340, using the detected next peak as the second peak, recording its peak value, valley value, peak position and valley position, and judging whether the peak energy difference ratio and peak-to-peak distance between the first peak and the second peak are greater than the preset The threshold value, if yes, execute step 350, otherwise, execute step 380;

The peak energy difference ratio between the first peak and the second peak is calculated as follows: the difference between the peak value of the second peak minus the valley value of the first peak is divided by the difference between the peak value of the first peak minus the valley value of the first peak, and the obtained The ratio is the peak energy difference ratio of the two peaks. When the peak energy difference ratio is smaller than the corresponding threshold, it is considered that the current second peak is a small spike that appears in the descending section of the first peak and does not correspond to a musical note. The threshold used for comparison with the calculated peak energy difference ratio is preferably 0.1825-0.3125, and this embodiment takes 0.1875. the

The peak-to-peak distance between two peaks refers to the number of speech frames between two peak positions, and the threshold used for comparison with the calculated number of speech frames is preferably 5-9, and it is 7 in this embodiment. If the peak-to-peak distance is smaller than the threshold, it means that the second peak is very close to the first peak, and it is not considered that the first peak and the second peak respectively correspond to a single note. the

Of course, the above two conditions, that is, the peak energy difference ratio and the peak-to-peak distance can also be used alone, or other double peak detection conditions can be used to filter out transition components. the

Step 350, continue to detect, judge whether to detect next wave peak before detecting the speech frame of energy less than critical value, if not, represent that there is no wave peak before the end of this voiced speech segment, execute step 360, if detect next wave peak, Execute step 370;

Step 360, determine that the first peak and the second peak correspond to a note respectively, output the position of the first peak and valley value and the end position of the voiced sound segment, and end;

Step 370, determine that the first peak corresponds to a note, output its valley position, and replace the saved parameters of the first peak with the parameters of the second peak at the same time, that is, the original second peak is used as the new first peak , return to step 340;

Step 380, continue to detect, judge whether to detect next wave peak before detecting the speech frame of energy less than critical value, if not, represent that there is no wave peak before the end of this voiced sound segment, execute step 390, if detect next wave peak, Execute step 400;

Step 390, determine that the first wave peak corresponds to a note, output the end position of the voiced sound segment, end;

Step 400, update the parameters of the first peak according to the parameters of the first peak and the second peak of the current record, and return to step 340;

When updating the first peak parameter, in this embodiment, the peak value of the first peak is replaced by the larger value of the first peak value and the second peak value, and one valley value or two valley values after the large peak value are used. The small value of replaces the valley value of the first peak, and the peak position and valley position of the first peak are updated to the speech frames corresponding to the new peak value and valley value. the

After the detection is completed or during the detection process, note segmentation can be performed according to the output voiced segment start, end position and peak valley position. On a voiced segment, two adjacent positions of the output are a note start and end positions. the

In general, there are always some transition components between two peaks, including small spikes and gentle declines (belonging to the continuation of the first peak), jitter and irregular rising curves (belonging to the start of the second peak), etc. , using double-peak detection to effectively deal with transition components, thus improving the accuracy of traditional peak detection algorithms. the

The present embodiment adopts the note segmentation device as shown in Figure 7 to realize the method for above-mentioned note segmentation, including a first-order low-pass filter (this unit is optional), a peak detection module, a main control module, a note segmentation module, Storage module and double-peak determination module, wherein:

The first-order low-pass filter is used to filter the energy of each speech frame in the detected speech frame sequence, and the peak detection module and the main control module process based on the filtered speech frame energy;

The peak detection module is used to count the voice frames whose energy involved in the energy continuous rising segment and the subsequent energy continuous falling segment from the initial voice frame of the voiced segment or the last voice frame of the previous energy falling segment is greater than the critical value number, if the number is greater than the third threshold, then it is determined that a section of energy curve corresponding to these speech frames constitutes a peak, the maximum energy value on this section of curve is the peak value of this wave peak, and the energy of the last speech frame on this section of curve The value is the valley value of the peak, and the peak and valley positions of the peak are the speech frames corresponding to the peak and the valley;

The storage module is used to save the parameters of the peak and the starting and ending positions of the voiced segment;

The double peak determination module is used to determine whether the ratio of the difference between the peak value of the second peak and the valley value of the first peak and the difference between the peak value of the first peak and the valley value is greater than the first threshold, and whether the peak value of the first peak and the second peak Whether the number of speech frames between the positions is greater than the second threshold, if they are all, the returned judgment result is success, otherwise the judgment result of failure is returned;

The main control module is used to segment each voiced sound segment formed by continuous speech frames with energy greater than a critical value, and further includes a first control unit, a second control unit, a third control unit, a fourth control unit and a first control unit. Five control units, of which:

The first control unit is used to call the peak detection module from the starting position of the voiced sound segment, and if no peak is detected, then end the processing of the voiced sound segment, otherwise the first peak detected is the first peak, and the The starting position of the voiced sound segment and the peak value, valley value, peak position and valley position of the first wave peak are stored in the storage module, triggering the second control unit to continue processing;

The second control unit is used to call the peak detection module. If the next peak is detected before the speech frame whose energy is less than the critical value is detected, the starting position of the voiced segment is output to the note segmentation module, and the third control unit is triggered to continue. Processing, otherwise, output the start and end positions of the voiced segment to the note segmentation module;

The third control unit uses the detected next peak as the second peak, records its peak value, valley value, peak position and valley position, calls the double peak determination module, and if the returned determination result is successful, triggers the fourth control unit Continue processing, otherwise trigger the fifth control unit to continue processing;

The fourth control unit is used to call the peak detection module, such as detecting the next peak before detecting the speech frame with energy less than the critical value, outputting the position of the first peak and valley to the note segmentation module, and using the parameter of the second peak Replace the corresponding parameters of the saved first peak, and trigger the third control unit to continue processing; otherwise, output the position of the first peak and valley and the end position of the voiced segment to the note segmentation module;

The fifth control unit is used to call the peak detection module, such as detecting the next peak before detecting the speech frame with energy less than the critical value, updating the parameters of the first peak according to the parameters of the first peak and the second peak (see Figure 6 In step 370), the third control unit is triggered to continue processing; otherwise, the voiced segment end position is output to the note segmentation module;

The note segmentation module is used to complete the note segmentation of the voiced segment by using two adjacent positions output during the processing of the voiced segment as the start and end positions of a note. the

Sometimes there will be some small spikes between the peaks. In the embodiment, the valley position of the previous peak is used as the end position of the note corresponding to the peak. However, in other implementation manners, the initial position of the next peak can also be used as the end position of the note corresponding to the previous peak, and the process of this reaction will record the initial position of the second peak in step 340, and in step 360 In step 370, first replace the valley position of the first peak with the initial position of the second peak, and then output the replaced valley position of the first peak. Reflected on the device, the third control unit also records the initial position of the second peak, and the fourth control unit first replaces the valley position of the first peak with the initial position of the second peak, and then outputs the replaced The valley position of the first peak. the

Step 30 in the overall process also converts the split notes into MIDI scores, the method is as follows:

As we all know, "1, 2, 3, 4, 5, 6, 7--do, re, mi, fa, so, la, xi" in each scale in numbered notation corresponds to a frequency value, and according to music theory The twelve equal laws correspond to different MIDI values, for example:

Octave5 (Octave) Midi Pitch

415.30HZ G5#So5# ---MIDI 68

440.00HZ A5 La5 ---MIDI 69

466.16HZ A5#La5# ---MIDI 70

MIDI notation requires note information, including the length of the note and the mean pitch of the note. the

The length of the note is obtained according to the number of speech frames between two adjacent positions output when the voiced segment is divided into notes. These positions may be the peak start position, the valley position, or the start and end of the voiced segment Location. the

The average value of the pitch of the note is obtained by finding the corresponding pitch curve at two adjacent positions output when the note is segmented for the voiced segment, and calculating the average value of the pitch of the curve. For example, the two adjacent positions of the output are 10, 35 (expressed by the speech frame number), and P(n) is the pitch value of the nth speech frame in the pitch curve, then the pitch mean of the note is:

Pitch＝[P(10)+P(11)+...+P(35)]/(35-10)

Convert the mean value of the pitch to the corresponding frequency f _pitch =f _x /Pitch, where f _x is the sampling frequency.

Therefore, first obtain the corresponding frequency value (or scale Octave value) through the pitch of the note, then quantize it to "1, 2, 3..." in the numbered notation, and finally use the twelve equal temperament formula MIDI=69 +12×log ₂ [(FS/440)×f _pitch ] to get the MIDI pitch. For example, if the system gets a note with a pitch of 430HZ, it will first be quantized into A5 La5 in Octave5, and then it can be expressed in MIDI 69 calculated by the twelve equal-tempered formula in advance.

In summary, the present invention can be applied to PCs and embedded systems, can track human voice humming signals and some electronic musical instruments, through time domain autocorrelation pitch detection (Pitch Detection) algorithm and energy-based note segmentation to The system occupies less resources, the method is simple, and the use is convenient and flexible. the

Claims (10)

1. pitch tracking and Play System, comprise: be used to receive the sound of input and it is done the phonetic entry processing module of sampling and the processing of branch frame, be used to detect the pitch and the energy detection module of the pitch and the energy of each speech frame, be used for finishing the note cutting device of note cutting according to energy detection results, and the note that is used for finishing cutting converts the music score modular converter of music score to, it is characterized in that, this system also comprises the phonetic synthesis module, and the music score that is used for converting to synthesizes musical instrument digital interface file and plays.

2. pitch tracking as claimed in claim 1 and Play System, it is characterized in that, described pitch and energy detection module further comprise pretreatment unit, Normalized Cross Correlation Function computing unit, post-processing unit and fundamental frequency search unit, wherein, this pretreatment unit adopts two rank low-pass filters to carry out filtering.

3. pitch tracking as claimed in claim 2 and Play System is characterized in that, the calculating that described cross correlation function computing unit carries out cross correlation function is that dot interlace carries out.

4. as claim 2 or 3 described pitch tracking and Play Systems, it is characterized in that described fundamental frequency search unit is taked the dot interlace search when search, and its hunting zone is within 20-120.

5. pitch tracking as claimed in claim 1 and Play System is characterized in that, described note cutting device comprises crest detection module, main control module, note cutting module, memory module and bimodal determination module, wherein:

Described crest detection module, be used to add up from the number of the related energy of continuous ascent stage of energy that the initial speech frame of voiced segments or last last speech frame of energy decreases section begin and the continuous descending branch of energy subsequently greater than the speech frame of critical value, as this number greater than the 3rd threshold value, one section energy trace then judging these speech frame correspondences constitutes a crest, energy value maximum on this section curve is the peak value of this crest, the energy value of last speech frame is the valley of this crest on this section curve, and the peak value of this crest and valley position are respectively the speech frame of this peak value and this valley correspondence;

Described memory module is used to preserve the starting and ending position of the parameter and the voiced segments of crest;

Described bimodal determination module is used to judge that whether the ratio of difference of the peak value of the difference of secondary peak peak value and primary peak valley and primary peak and valley is greater than first threshold, and whether the speech frame number between the peak of primary peak and secondary peak is greater than second threshold value, if all be, the result of determination of then returning is successfully, otherwise returns the result of determination of failure;

Described main control module is used for energy is carried out the note cutting greater than each voiced segments of the continuous speech frame formation of critical value, further comprise first control module, second control module, the 3rd control module, the 4th control module and the 5th control module, wherein:

First control module, be used for reference position from voiced segments, call the crest detection module, as detecting less than crest, then finish the processing of this voiced segments, otherwise with detected first crest is primary peak, and peak value, valley, peak and the valley position of this voiced segments reference position and primary peak is saved in described memory module, triggers second control module and continues to handle;

Second control module, be used to call the crest detection module, as before detecting the speech frame of energy, detecting next crest less than critical value, then the voiced segments reference position is outputed to note cutting module, triggering the 3rd control module continues to handle, otherwise, this voiced segments starting and ending position is outputed to note cutting module;

The 3rd control module, with detected next crest as secondary peak, write down its peak value, valley, peak and valley position, call the bimodal determination module, if the result of determination of returning is successfully, trigger the 4th control module and continue to handle, continue to handle otherwise trigger the 5th control module;

The 4th control module, be used to call the crest detection module, as before detecting the speech frame of energy, detecting next crest less than critical value, primary peak valley position is outputed to note cutting module, and override the relevant parameter of the primary peak of preservation with the parameter of secondary peak, trigger the 3rd control module and continue to handle; Otherwise primary peak valley position and voiced segments end position are outputed to note cutting module;

The 5th control module, be used to call the crest detection module, as before detecting the speech frame of energy, detecting next crest less than critical value, replace the peak value of primary peak with the big value in primary peak and the secondary peak peak value, replace the valley of primary peak with the minimum value in valley behind this big peak value or two valleies, the peak of primary peak and valley position renewal are the new peak value and the speech frame of valley correspondence, trigger the 3rd control module then and continue to handle; Otherwise the voiced segments end position is outputed to note cutting module;

The note cutting to this voiced segments is finished in the starting and ending position that two adjacent positions that described note cutting module is used for exporting with the voiced segments processing procedure are a note.

6. pitch tracking and player method may further comprise the steps:

(a) sound import is done sampling and divided frame to handle;

(b) detect the pitch and the energy of each speech frame;

(c) finish the note cutting according to energy detection results, convert note cutting result to music score according to detected pitch then;

(d) music score that is converted to is synthesized musical instrument digital interface file, and play.

7. pitch tracking as claimed in claim 6 and player method is characterized in that, when described step (b) detects the pitch of speech frame, are further divided into following steps:

(b1) voice signal after minute frame processing is carried out pre-service, comprise average and low-pass filtering treatment and adopt two rank low-pass filters to carry out filtering;

(b2) pretreated speech energy and a threshold value are compared, as be higher than this threshold value, then be judged as voiced sound, forward step (b3) to, otherwise, execution in step (b7);

(b3) calculate the Normalized Cross Correlation Function value ρ (t) of the voiced sound signal of every frame delay;

(b4) carry out aftertreatment, obtain the ρ (t) and the corresponding optimal delay T of correlativity maximum;

(b5) fundamental frequency search, the sampling point t of the pairing time delay of ρ (t) that promptly detects the correlativity maximum whether in the hunting zone, if in this scope, execution in step (b6), otherwise, execution in step (b7);

(b6) think that these frame voice are voiced sound, output optimal delay T, according to the relation of Time And Frequency, the pitch value by this T converses this frame voice correspondence finishes again;

(b7) think that these frame voice are voiceless sound, make that pitch is 0, finish.

8. pitch tracking as claimed in claim 7 and player method is characterized in that, described step (b3) is that dot interlace carries out when carrying out the calculating of cross-correlation function value.

9. pitch tracking as claimed in claim 7 and player method is characterized in that, described step (b5) is taked the dot interlace search when carrying out the fundamental frequency search, and its hunting zone is within 20-120.

10. as claim 6 or 8 described pitch tracking and player methods, it is characterized in that, when described step (c) is carried out the note cutting,, carry out following steps energy each voiced segments greater than the continuous speech frame formation of critical value:

(c1) with first speech frame be the reference position of this voiced segments, as detect crest, with first detected crest as primary peak, write down peak value, valley, peak and the valley position of this voiced segments reference position and primary peak, carry out next step, as detecting, finish less than crest;

(c2) continue to detect,, then export the voiced segments reference position as before detecting the speech frame of energy, detecting next crest less than critical value, execution in step (c3), otherwise export this voiced segments starting and ending position, execution in step (c6);

(c3) with detected next crest as secondary peak, write down its peak value, valley, peak and valley position, the ratio of difference that judges whether the peak value of the difference of secondary peak peak value and primary peak valley and primary peak and valley is greater than first threshold, and the speech frame number between the peak of primary peak and secondary peak is greater than second threshold value, if, execution in step (c4), otherwise execution in step (c5);

(c4) continue to detect, as before detecting the speech frame of energy, detecting next crest less than critical value, output primary peak valley position, and override the relevant parameter of the primary peak of preservation with the parameter of secondary peak, return step (c3), otherwise, output primary peak valley position and voiced segments end position, execution in step (c6);

(c5) continue to detect, as before detecting the speech frame of energy, detecting next crest less than critical value, override the peak value of primary peak with the big value in primary peak peak value and the secondary peak peak value, override the valley of primary peak with the little value in valley behind this big peak value or two valleies, the peak of primary peak and valley position renewal are the new peak value and the speech frame of valley correspondence, return step (c3) then, otherwise, export this voiced segments end position, execution in step (c6);

(c6) with the starting and ending position that is a note, two adjacent positions of output, finish note cutting to this voiced segments.

Wherein, when described step (a)～(e) is carried out the crest detection, it is the energy size of more adjacent one by one two speech frames, statistics is from the number of the related energy of continuous ascent stage of energy that the initial speech frame of voiced segments or last last speech frame of energy decreases section begin and the continuous descending branch of energy subsequently greater than the speech frame of critical value, as this number greater than the 3rd threshold value, one section energy trace then judging these speech frame correspondences constitutes a crest, energy value maximum on this section curve is the peak value of this crest, the energy value of last speech frame is the valley of this crest on this section curve, the peak value of this crest and valley position are respectively the speech frame of this peak value and this valley correspondence, and the initial speech frame of this voiced segments or last last speech frame of energy decreases section are the reference position of this crest.

Images