CN111785296A - A music segmentation boundary recognition method based on repetitive melody - Google Patents

Abstract

The invention relates to a music segmentation boundary identification method based on repeated melody, belonging to the technical field of audio signal processing. The method comprises the following steps: 1) extracting chroma characteristics from the audio, filling zero from beginning to end, aggregating every adjacent N frames to form a new frame vector, and forming a new frame characteristic vector sequence by all the frame vectors; 2) calculating Euclidean distance between each frame vector and other frame vectors in the frame feature sequence to obtain a self-similarity matrix S; 3) based on the self-similarity matrix S, obtaining a set N of the ith frame vector nearest neighbor frame_iAnd obtaining a recursion graph R of the self-similarity matrix S; 4) carrying out time delay processing on the recursive graph R to obtain a time delay matrix L; 5) carrying out line segment normalization and denoising on the L, and then carrying out reverse time delay processing to obtain a recursion graph R'; 6) and detecting all line segments, clustering the line segments, and sequentially processing from the cluster with the largest line segment to obtain a music segmentation boundary point set B. The recognition capability of the repeated melody in the music can be improved, and the music can be segmented in a shorter time.

Description

A music segmentation boundary recognition method based on repetitive melody

technical field

The present invention relates to the technical field of audio signal processing, in particular to a method for identifying boundaries of music segments based on repeated melody.

Background technique

Information is often organized in a structure or hierarchy to facilitate dissemination or understanding. Humans are generally very good at perceiving such structures, sometimes even unconsciously, to allow us to analyze and fully grasp the meaning of a given information. However, considering the situation in the era of big data, we increasingly need to obtain information processing support from computers. Therefore, automating the structure of obtaining information becomes a critical task of today's content processing systems. Among a wide range of multimedia content, music is a typical example.

Important applications of music segmentation boundary recognition algorithm research include player navigation within works, automatic generation of segments and mashups, identification of versions of identical works, and large-scale musicology research. With the popularization and development of network and digital entertainment products, music has become one of the most important digital media contents.

At present, in addition to being an independent entertainment product, music also plays an important role in the form of soundtrack in film and television works. As an independent entertainment product, music segmentation is an important basic process in music analysis. In the analysis scenario of a certain type of music works, the huge number of works highlights the importance of automatic music segmentation. As a soundtrack, in practical applications, rather than the whole piece of music, the fragment is used in more cases. Automatic music segmentation can greatly improve the efficiency of music fragment extraction. It can be seen that the research of music segmentation boundary recognition algorithm has broad market application prospects.

In 2000, Foote first used self-similarity matrices in music segmentation algorithms to discover repetitive melodies in music. A 2006 study by Bruderer et al. pointed out that there are cues that are highly relevant to human perception of musical structure, such as timbre changes, repetitions, and pauses. A 2010 study by Paulus et al. states that there are three principles for inferring musical structure: novelty, homogeneity, and repetition. The music segmentation algorithm proposed by Serra et al. in 2014 comprehensively considered these principles and introduced the calculation method of the recursive graph, which greatly improved the segmentation accuracy, thereby improving the efficiency of automatic music segmentation and promoting the automatic music segmentation algorithm. development of.

However, there are still many shortcomings in the algorithms currently applied to music segmentation. For example, the segmentation granularity of the unsupervised method is relatively large, and it is difficult to obtain short segments of some music. Problems with mathematical methods. Deep learning methods fail to fully consider the nature of repetition in segmentation, and have problems of dependence on data, high model training costs, and difficulty in integrating music theory knowledge.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for identifying the boundaries of music segments based on repetitive melody, so as to improve the ability to recognize repetitive melody in music and to segment music on a shorter time scale.

In order to achieve the above object, the method for identifying the boundaries of music segments based on repeated melody provided by the invention comprises the following steps:

1) extracting chroma features from the audio to obtain a sequence of feature vectors, which is a total of M frames; zero-fill the beginning and end of the sequence of eigenvectors, aggregate every adjacent N frames to form a new frame vector, and all frame vectors form a new sequence of frame eigenvectors;

2) Calculate the Euclidean distance between each frame vector and other frame vectors in the frame feature sequence to obtain a self-similar matrix S;

3) Based on the self-similar matrix S, obtain the set N _i of the nearest neighbor frame of the ith frame vector, i=1, 2, ..., M, and obtain the recursive graph R of the self-similar matrix S accordingly;

4) The recursive graph R is processed by time delay to obtain a time delay matrix L;

5) Carry out line segment regularization and denoising to the time delay matrix L, and then inverse time delay processing to obtain a recursive graph R' after regularization and denoising;

6) Based on the recursive graph R', detect all the line segments and perform line segment clustering, and process sequentially from the cluster with the most line segments to obtain the music segment boundary point set B.

In the above-mentioned technical scheme, for the repeated segments of music, the pitch class profile (PitchClass Profile) feature of the music is extracted in frames, also called the Chroma feature, which organizes the frequencies of a given range into 12 pitch classes, Highlight the melody of the music.

Optionally, in one embodiment, in step 3), for the k elements in the set N _i are the k frame vectors most similar to the ith frame vector in all frame vectors, and the value of k is the frame vector 0.01 of the total. For each point R _i,j in the recursive graph R, if i belongs to N _j and j belongs to N _i , take R _i,j equal to 1, otherwise take R _i,j equal to 0, and thus obtain the self-similar matrix S The recursive graph R of .

Optionally, in an embodiment, in step 4), let _Li,j =R _{i,(i+j)mod(M-1)} , i=1, 2, . . . , M, j=1, 2, ..., M, obtain the time delay matrix L of the recursive graph R, that is, convert the main diagonal direction in the recursive graph R to the horizontal direction.

Optionally, in one embodiment, step 5) includes:

5-1) Traverse the time delay matrix L, and the value of 1 is defined as a point; every time a point is found, all points connected to it are determined by breadth-first search, and the step distance is less than 3, it is considered to be connected;

5-2) Count the number of points with the same ordinate in the connected points. If the number of points under the ordinate with the largest number of points is greater than 5, the point of the ordinate among these points is retained, and the other points are set to 0 ; otherwise, all these points are set to be 0;

5-3) Let R' _{i,(i+j)mod(M-1)} =L _i,j , i=1,2,...,M,j=1,2,...,M, get the regularization and removal The noised recursive graph R'.

Optionally, in one embodiment, in step 6), the line segment clustering includes:

Traverse the recursive graph R' and set the stride to 3.

Find all the line segments in the figure, and use {x ₁ , x ₂ , y ₁ , y ₂ } to standardize each line segment, where x ₁ and x ₂ are the abscissas of the start and end points, and y ₁ and y ₂ are the ordinates of the start and end points. ;

Take a line segment, traverse other line segments, find all the line segments corresponding to the same melody and cluster them; the basis for judging that they correspond to the same melody is: the common length of x ₁ and x ₂ accounts for more than 80% of each.

和

然后针对该簇中的每条线段，根据x₁和

x₂和

的差值分别对y₁和y₂进行修正得到y’₁和y’₂；将

和所有的y’₁、y’₂作为时间点x进行如下处理：检查音乐分段边界点集合B中，是否存在与时间点x相隔小于n帧的分段边界点，若不存在则将时间点x加入B中。Optionally, in one embodiment, in step 6), after the line segments are clustered, the cluster with the largest number of line segments is taken, and the average value of all x ₁ and x ₂ is obtained to obtain:

and

Then for each line segment in that cluster, according to x ₁ and

x ₂ and

The difference of y ₁ and y ₂ are respectively corrected to obtain y' ₁ and y'₂;

And all y' ₁ and y' ₂ are processed as time point x as follows: check whether there is a segment boundary point that is less than n frames away from time point x in the set B of music segment boundary points, if not, set the time Click x to join B.

Compared with the prior art, the advantages of the present invention are:

The present invention uses music theory knowledge and practical experience to perform matrix denoising, fully considers the main reasons for noise generation in music segmentation, and can reduce errors caused by noise more thoroughly and efficiently. However, the segmentation point acquisition method based on line clustering gives priority to the melody segment with many repetitions, and the method of taking the average value as the segmentation point further reduces the error and improves the generalization performance.

Description of drawings

Fig. 1 is the flow chart of the music segmentation boundary identification method based on repeated melody in the embodiment of the present invention;

2 is a schematic diagram of a recursive graph R in an embodiment of the present invention;

3 is a schematic diagram of a delay matrix L in an embodiment of the present invention;

Fig. 4 is a recursive graph R' after regularization and denoising in an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described below with reference to the embodiments and the accompanying drawings. Obviously, the described embodiments are some, but not all, embodiments of the present invention. Based on the described embodiments, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Unless otherwise defined, technical or scientific terms used in the present invention should have the ordinary meaning as understood by one of ordinary skill in the art to which the present invention belongs. When used herein, "comprises" or "comprising" and similar words mean that elements or steps appearing before the word encompass the elements or steps listed after the word and their equivalents, but do not exclude other elements or steps.

Example

This embodiment constructs a music segmentation algorithm based on a self-similar matrix based on a method for identifying the boundaries of music segments based on repeated melody, and realizes automatic identification of music structure segment points. This method can replace manual annotation for the generation of music structure sequences, and can be further applied to music analysis, automatic segment generation, etc. Referring to Figure 1, the specific process is as follows:

S100, extract chroma features from the audio to obtain a sequence of feature vectors, totaling M frames; zero-pad the beginning and end of the sequence of feature vectors, aggregate every adjacent N frames to form a new frame vector, and all frame vectors form a new sequence of frame feature vectors;

The feature sequence of the sample music is a 12-dimensional vector sequence with a length of 1344. The first and last zero-padding obtains a length sequence with a length of 1350, and each adjacent 7 frames are aggregated to form a new frame feature sequence, and a 12x7-dimensional vector sequence is obtained, and the length is still 1344.

S200 , calculating the Euclidean distance between each frame vector and other frame vectors in the frame feature sequence, to obtain a self-similar matrix S of 1344×1344.

S300, based on the self-similar matrix S, obtain the set N _i of the nearest neighbor frame of the ith frame, i=1, 2, ..., M, and obtain the recursive graph R of the self-similar matrix S accordingly, see Fig. 2;

The k elements in set Ni are the k frames most similar to the _i -th frame among all frames. For each point R _i,j in the recursive graph, if i belongs to N _j and j belongs to N _i , take Ri _,j equal to 1, otherwise take Ri _,j equal to 0, and get a recursive graph R of 1344x1344. The value of k is 0.01 of the total number of frames, which is 13 in this embodiment.

S400, the recursive graph R is subjected to time delay processing to obtain a time delay matrix L, see FIG. 3;

_{Let Li,j} =R _{i,(i+j)mod(M-1) first} , get the time delay matrix L of the recursive graph R, convert the main diagonal direction in the recursive graph R to the horizontal direction, and improve the computational efficiency .

S500, performing line segment regularization and denoising on the time delay matrix L, and then inverse time delay processing to obtain a regularized and denoised recursive graph R', see Fig. 4 .

First, the time delay matrix L is traversed, and the value of 1 is defined as a point. Each time a point is found, all points connected to it are determined by breadth-first search, and the step distance is less than 3, it is considered to be connected. Count the number of points with the same ordinate in each of these connected points. If the number of points on the ordinate with the largest number of points is greater than 5, keep the point of this ordinate among these points, and take 0 for other points. Otherwise, the values of these points are all 0. For example, a series of points is {(1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(2,2),( 3,2). Then, let R' _{i,(i+j)mod(M-1)} =L _i,j to obtain the recursive graph R' after regularization and denoising.

S600, based on the recursive graph R', detect all line segments and perform line segment clustering, and process sequentially from the cluster with the most line segments to obtain a music segment boundary point set B.

First find all the line segments in the recursive graph R' and standardize the representation, traverse the recursive graph R', set the stride to 3, and find all the line segments. After the line segment is found, it is represented by {x ₁ , x ₂ , y ₁ , y ₂ }, where x ₁ and x ₂ are the abscissas of the starting and ending points, and y ₁ and y ₂ are the ordinates of the starting and ending points. For example, a line segment is {1,9,10,19}, which means that frames 10 to 18 are similar to frames 1 to 9. Then, in all line segments, the common parts of x ₁ and x ₂ account for more than 80% of their respective parts, clustered in the same cluster, such as {1, 9, 10, 18}, {2, 9, 20, 27} and {2, 9} ,31,38}. After clustering, take the average value of x ₁ and mark the corresponding y _1. For example, the average value of x ₁ here is 2, and the corresponding 3 y ₁ will be 11, 20 and 31. Check whether there are points in the boundary point set B that are within 20 frames (related to the required segmentation time) and add them to B if they do not exist. In this way, the segmentation result of the sample music is obtained.

Claims (8)

1. a kind of music segmentation boundary recognition method based on repeated melody, is characterized in that, comprises the following steps: 1) extracting chroma features from the audio to obtain a sequence of feature vectors, which is a total of M frames; zero-fill the beginning and end of the sequence of eigenvectors, aggregate every adjacent N frames to form a new frame vector, and all frame vectors form a new sequence of frame eigenvectors; 2) Calculate the Euclidean distance between each frame vector and other frame vectors in the frame feature sequence to obtain a self-similar matrix S; 3) Based on the self-similar matrix S, obtain the set N _i of the nearest neighbor frame of the ith frame vector, i=1, 2, ..., M, and obtain the recursive graph R of the self-similar matrix S accordingly; 4) The recursive graph R is processed by time delay to obtain a time delay matrix L; 5) Carry out line segment regularization and denoising to the time delay matrix L, and then inverse time delay processing to obtain a recursive graph R' after regularization and denoising; 6) Based on the recursive graph R', detect all the line segments and perform line segment clustering, and process sequentially from the cluster with the most line segments to obtain the music segment boundary point set B. 2. the music segmentation boundary identification method based on repeated melody according to claim 1, is characterized in that, in step 3), for k elements in set N _i is the most with the i-th frame vector in all frame vectors. Similar k frame vectors, the value of k is 0.01 of the total number of frame vectors. 3. the music segmentation boundary identification method based on repeated melody according to claim 1, is characterized in that, in step 3), for each point R i in the recursive graph R _i,j , if i belongs to N _j and j If it belongs to N _i , take Ri _,j equal to 1, otherwise take Ri _,j equal to 0, and thus obtain the recursive graph R of the self-similar matrix S. 4. the music segmentation boundary identification method based on repeated melody according to claim 1, is characterized in that, in step 4), make _Li,j =R _{i,(i+j)mod(M-1)} , i=1, 2, . 5. the music segmentation boundary identification method based on repeated melody according to claim 4, is characterized in that, step 5) comprises: 5-1) Traverse the time delay matrix L, and the value of 1 is defined as a point; every time a point is found, all points connected to it are determined by breadth-first search, and the step distance is less than 3, it is considered to be connected; 5-2) Count the number of points with the same ordinate in the connected points. If the number of points under the ordinate with the largest number of points is greater than 5, the point of the ordinate among these points is retained, and the other points are set to 0 ; otherwise, all these points are set to be 0; 5-3) Let R' _{i,(i+j)mod(M-1)} =L _i,j , i=1,2,...,M,j=1,2,...,M, get the regularization and removal The noised recursive graph R'. 6. the music segmentation boundary recognition method based on repeated melody according to claim 1, is characterized in that, in step 6), line segment clustering comprises: Traverse the recursive graph R', find all the line segments in the graph, and use {x ₁ , x ₂ , y ₁ , y ₂ } to standardize each line segment, x ₁ and x ₂ are the abscissas of the starting and ending points, y ₁ and y ₂ is the ordinate of the start and end points; Take a line segment, traverse other line segments, find all the line segments corresponding to the same melody and cluster them; the basis for judging that they correspond to the same melody is: the common length of x ₁ and x ₂ accounts for more than 80% of each. 7. the music segmentation boundary identification method based on repeated melody according to claim 6, is characterized in that, when traversing recursive graph R ', setting step distance is 3. 8. the music segmentation boundary identification method based on repeated melody according to claim 6, is characterized in that, in step 6), after line segment is clustered, get the cluster that line segment number is the most, all x ₁ and x ₂ are taken. average, get

and

Then for each line segment in that cluster, according to x ₁ and

x ₂ and

The difference of y ₁ and y ₂ are respectively corrected to obtain y' ₁ and y'₂;

and all y' ₁ and y' ₂ as the time point x, and perform the following processing: check whether there is a segment boundary point that is less than n frames away from the time point x in the music segment boundary point set B, and if not, the time Click x to join B.

Images