KR20200118587A - Music recommendation system using intrinsic information of music - Google Patents

Abstract

The present invention relates to a music recommendation system. The music recommendation system according to the present invention is a data collection module 100 that collects music of representative singers for each genre from a remote server, and removes the live performance version when a digital single version of the same music and a live performance version exist among the collected music. ; A highlight extraction module 200 for extracting highlights of the collected music; A feature extraction module 300 for extracting musical features from the extracted highlights of music; And a music classification module 400 for classifying the collected songs based on the extracted features among similar songs.

Description

Music recommendation system using intrinsic information of music {MUSIC RECOMMENDATION SYSTEM USING INTRINSIC INFORMATION OF MUSIC}

The present invention relates to a music recommendation system using intrinsic information of music, and more particularly, extracts a highlight portion of music, determines the similarity of music based on characteristics of melody data and beat data of the extracted highlight portion, and It relates to a music recommendation system that classifies and recommends.

Modern people who listened to music by downloading music files to MP3 players in the past listen to music through music streaming services such as Melon, Gini, Bugs, and Mnet as smartphones are popular and data usage fees are gradually falling. Streaming service providers are striving to provide music recommendation services through data analysis in order to provide users with a better experience and use their services. In the case of Melon's Fou U, if you enter 3 preferred artists, 5 songs of interest, and gender, recommended songs are analyzed by analyzing interest and similarity, and artists similar to the selected artist are also listed. Bugs' Music 4U and Music PD list playlists for the same keyword selected by the PDs.

Meanwhile, the music recommendation service may consist of a user-based recommendation system, an item-based recommendation system, and a hybrid system combining the two. This recommendation system is strategically used not only by music but also by various retailers such as Amazon, Auction, and Coupang. In the recommendation system, music or products are recommended based on user preferences or relevance of items, but technically, recommendations are made based on metadata. Metadata is indirect data that describes itself. For example, in the case of music, the composer, lyricist, singer, genre, album, year, etc. of the music may be metadata for the music. However, since this metadata is not based on the content of the item, but is indirectly assigned, it is not lacking in expressing the content of the item depending on the situation.

Since the conventional music recommendation service performs user-based and item-based recommendations based on the metadata of specific music, it cannot recommend unpopular music for which sufficient data is not secured, and does not recommend based on the intrinsic data of the actual music. There is a limit to not being able to. In other words, there is a limitation in that no conventional metadata can express the inherent data of music.

Korean Patent Registration No. 10-0895009 Korean Patent Registration No. 10-0955523

An object of the present invention for solving the above-described problem is to a music recommendation system that improves the efficiency of determining similarity by extracting a highlight portion, not the entire section of music.

Another object of the present invention relates to a music recommendation system that improves the accuracy of determining the similarity of music by extracting features of a melody side and a feature of a beat side inherent in a highlight portion of music.

Another object of the present invention relates to a music recommendation system that extracts MFCC characteristics, chroma characteristics, and tempo characteristics in terms of a melody and a beat of music, and improves the accuracy of determining the similarity of music by using them.

In order to achieve the above object, the music recommendation system according to an embodiment of the present invention collects music of representative singers for each genre from a remote server, and there is a digital single version and a live performance version of the same music among the collected music. A data collection module 100 to remove the live performance version; A highlight extraction module 200 for extracting highlights of the collected music; A feature extraction module 300 for extracting musical features from the extracted highlights of music; And a music classification module 400 for classifying the collected songs based on the extracted features among similar songs.

Preferably, the highlight extraction module 200 divides music into an intro, a verse, a chorus, a bridge, and an outro using a Musical Structural Analysis Framework (MSAF) technique. And a section classification module 210 for identifying by attaching a label to each divided section, and the highlight extraction module 200 extracts a chorus and a repetition phrase from the music using a label, and then extracts the chorus and It may include a highlight determination module 220 that determines the chorus and repetition phrases that are most frequently repeated in the music among the repetition phrases, and determines a section appearing at the earliest part of the music among the determined chorus and repetition phrases as a highlight.

Preferably, the highlight determination module 220 does not determine the chorus and repetition phrase as highlights when the chorus and repetition phrase determined as the highlight exist in front of the 1/4 point of the music, but the next most repeated chorus and Among the repeating phrases, a chorus and a repeating phrase present after a quarter point of the music may be determined as highlights.

Preferably, the feature extraction module 300 is a Mel-Filtering module 310 that separates and extracts melody data and bit data included in the music highlight, and DCT (Discrete Cosine Transform) an audio signal corresponding to the music highlight. An MFCC extraction module 320 that processes and extracts MFCC (Mel-Frequency Cepstral Coefficient) coefficients, and an audio signal corresponding to the highlight of music is converted to Short-Time Fourier Transform (STFT), Constant Q Transform, and Chroma-Energy Normalized Statistics (CENS). ), the tempo may be calculated through the chroma extraction module 330 that extracts the chroma feature by processing and the bit data separated and extracted from the highlight.

Preferably, the music classification module 400 is a K-means module 410, a Gaussian Mixture Model (GMM) for classifying similar music by a K-means technique based on features extracted by the feature extraction module 300 A GMM module 420 for classifying similar music by a technique and a DBSCAN module 430 for classifying similar music by a Density-based spatial clustering of applications with noise (DBSCAN) technique.

The present invention can provide a music recommendation system that improves the efficiency of determining similarity by extracting a highlight portion, not the entire section of music.

The present invention can provide a music recommendation system that improves the accuracy of determining the similarity of music by extracting the features of the melody side and the features of the beat side inherent in the highlight portion of music.

The present invention can provide a music recommendation system that extracts MFCC characteristics, chroma characteristics, and tempo characteristics in terms of the melody and the beat of music, and improves the accuracy of determining the similarity of music by using them.

1 is a block diagram showing the configuration of a music recommendation system according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a highlight extraction module according to an embodiment of the present invention.
3 is a block diagram showing the configuration of a feature extraction module according to an embodiment of the present invention.
4 is a block diagram showing the configuration of a music classification module according to an embodiment of the present invention.
5 is a block diagram schematically illustrating a feature extraction process of a feature extraction module according to an embodiment of the present invention.
6 is a diagram showing a graph showing a BIC value according to the number of clusters used in a classification process of a music classification module according to an embodiment of the present invention and a list of classified music.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding reference numerals to constituent elements in each drawing, it should be noted that the same constituent elements are given the same reference numerals as much as possible even though they are indicated on different drawings.

Further, in describing an embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with an understanding of the embodiment of the present invention, a detailed description thereof will be omitted.

In addition, in describing the constituent elements of the embodiments of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the component is not limited by the term.

A configuration and operation of a music recommendation system according to an embodiment of the present invention will be described below with reference to FIGS. 1 to 6.

1 is a block diagram showing the configuration of a music recommendation system (hereinafter referred to as “the system”) according to an embodiment of the present invention.

Referring to FIG. 1, the system includes a data collection module 100, a highlight extraction module 200, a feature extraction module 300, and/or a music classification module 400.

The data collection module 100 collects music of representative singers by genre from a remote server (such as YouTube). Redundant music among the collected music is removed, and if there is a digital single version and a live performance version of the same music among the collected music, the live performance version is removed.

The highlight extraction module 200 extracts highlights of the collected music. Since measuring the similarity based on the whole music is meaningless in terms of efficiency, only the highlight part of the music is extracted and used to determine the similarity. A detailed description of this module will be described later.

The feature extraction module 300 extracts musical features from the highlights of the music extracted by the highlight extraction module 200. The similarity of music is judged by extracting features that did not exist before and analyzing the features extracted by a new technique. A detailed description of this module will be described later.

The music classification module 400 classifies the collected music based on the feature extracted by the feature extraction module 300 among similar music, and finally determines the similarity between the collected music. A detailed description of this will be described later.

2 is a block diagram showing the configuration of a highlight extraction module according to an embodiment of the present invention.

Referring to FIG. 2, the highlight extraction module 200 includes a section division module 210 and/or a highlight determination module 220.

The section division module 210 divides music into intro, verse, chorus, bridge, and outro using a Musical Structural Analysis Framework (MSAF) technique, and Each section is identified by labeling it.

More specifically, most of the music is intro, verse, chorus, verse, chorus, bridge, verse, chorus, and It has an outro structure. In this structure, the weight of the chorus is the highest, and the listener recognizes the chorus as a major part of the music. Furthermore, the music includes a hook part, and in the case of hook songs, the proportion of the repeat phrase in the total music exceeds 40%.

The section classification module 210 classifies a section of the entire music using a MusicalStructural Analysis Framework (MSAF) package technique. The determination of which part of the sections divided by MSAF corresponds to a chorus or a repeat phrase can be made qualitatively. The section division module 210 may use “foote” to divide sections, and use a “scluster” function to label each section.

The highlight extraction module 200 extracts the chorus and repetition phrases from the music using the label, and then determines the chorus and repetition phrases that are most frequently repeated in the music among the extracted chorus and repetition phrases, and The section that appears at the beginning of the music is primarily determined as a highlight. Then, the highlight determination module 220 does not determine the chorus and repeat phrases as highlights when the chorus and repeating phrases that are primarily determined as highlights exist in front of the 1/4 point of the music, and repeats the next many times. Among the chorus and repeat phrases, the chorus and repeat phrases that exist after 1/4 of the music are determined as highlights.

More specifically, the highlight extraction module 200 determines a chorus and a repetition phrase among the labels (sections) classified and identified by the section division module 210 as highlights. In addition, since the chorus and the repetition phrase in the entire music have the characteristic that the most repetitive phrases are repeated, the highlight extraction module 200 determines the section of the label that is most frequently repeated in the entire music as a chorus or a repetition phrase and determines this as a highlight. Then, the chorus or repetition phrase shown at the beginning of the music is decided as the highlight. However, in this case, there may be a phenomenon in which the verse indicated before the chorus or the repeating phrase is regarded as a highlight, and to prevent this, if the repeated label is in front of the 1/4 point of the music circle, the label is not recognized as the highlight of the music. If a label that is repeated many times in sequential order is behind a quarter of the music, the section is selected as a highlight. Further, if the highlight cannot be determined even by the above-described process, a section appearing immediately after the 1/4 point of the music is arbitrarily determined as the highlight of the music. According to another embodiment, a quarter point for determining the highlight may be changed to a 1/8 point. In addition, when the duration of the portion determined as the highlight is less than 10 seconds, the highlight extraction module 200 does not determine the corresponding portion as the highlight, and determines the next-order section as the highlight as described above.

3 is a block diagram showing the configuration of a feature extraction module according to an embodiment of the present invention. 5 is a block diagram schematically illustrating a feature extraction process of a feature extraction module according to an embodiment of the present invention.

3 and 5, the feature extraction module 300 includes a Mel-Filtering module 310, an MFCC extraction module 320, a chroma extraction module 330 and/or a tempo calculation module 340.

The Mel-Filtering module 310 separates and extracts melody data and beat data included in the highlight of music. Specifically, the waveform data of the chorus or repetition section does not have a musical meaning by itself, but is a simple sequence of magnitude values of sounds. In order to express a song in the form of data having a musical meaning, it is necessary to extract and use auditory information and features related to music. To this end, the Mel-Filtering module 310 may use Python's Librosa package. The Mel-Filtering module 310 samples the audio signal for the highlight section at 22.05 kHz and classifies it in units of frames. Then, the audio signal corresponding to each frame is processed by a Hann window and converted into a frequency meter value through a short-time fourier transform (STFT) process. And, based on the converted value, a feature related to a melody and a feature related to a bit are separated and extracted. At this time, it is possible to separate the melody of the music from the percussion sound (beat) using the median-filtering harmonic-percussive separation algorithm provided by the Librosa package. Further, the Mel-Filtering module 310 processes the Mel Filter Bank to analyze the highlight section more closely, and applies a log scale to match the unit in a form in which humans perceive sound.

The MFCC extraction module 320 extracts a Mel-Frequency Cepstral Coefficient (MFCC) coefficient by processing the audio signal corresponding to the highlight of the music with Discrete Cosine Transform (DCT). For reference, since MFCC is based on a model of human vocalization, feature extraction is possible even in digital music in which sounds from various instruments included in music are combined.

The chroma extraction module 330 extracts a chroma feature by processing an audio signal corresponding to a highlight of a music with a short-time fourier transform (STFT), a constant Q transform, and a chroma-energy normalized statistics (CENS). Specifically, music is composed of 12 scales, and the intention is determined by extracting a chroma feature, which is a frequency component corresponding to the 12 scales. At this time, the chroma extraction module 330 extracts a chroma feature of the highlight using a Short-Time Fourier Transform (STFT), a Constant Q Transform, and/or a Chroma-Energy Normalized Statistics (CENS) technique.

The tempo calculation module 340 calculates a tempo through bit data separated and extracted from the highlight. Specifically, the tempo calculation module 340 extracts bit information from the percussive data classified by the Mel-Filtering module 310 and calculates a tempo using this.

As described above, the features extracted by the feature extraction module 300 are all expressed as vectors of the same length for distance calculation. This is because the Euclidean distance of each factor located in the space where the feature is expressed is calculated and the similarity is determined through the calculated distance. Further, if the length of the highlight section is different, the number of frames is also different, so the average of the feature values calculated for each frame is taken and used to determine the similarity.

4 is a block diagram showing the configuration of a music classification module according to an embodiment of the present invention. 6 is a diagram illustrating a graph showing a BIC value according to the number of clusters used in a classification process of a music classification module according to an embodiment of the present invention and a list of classified music.

4 and 6, the music classification module 400 includes a K-means module 410, a GMM module 420 and/or a DBSCAN module 430.

The K-means module 410 classifies similar music by the K-means technique based on the features extracted by the feature extraction module 300. The K-means method is an analysis method that uses the distance or similarity between given observations when there is no prior information about a population or category, and the analysis starts from determining the number of clusters. The K-means module 410 defines the number K of clusters and selects the center point of the initial K clusters. All observed values are assigned to the nearest central cluster, and the centroid of the new cluster is calculated. The distance-based cluster is reclassified based on the redefined center value, and the calculation is terminated when the boundary does not change, and the final cluster is formed.

The graph at the top left of FIG. 6 is a graph showing BIC values according to the number of clusters in order to find the most suitable K value for this embodiment. According to an embodiment, the K-means module 410 sets 3, 4, and 5, which are points where the BIC value in the graph sharply curves, as an initial K value, and K-means data consisting of 611 songs and 302 features. Music was classified by applying means technique. As a result, when K values were different, the songs closest to the median value were different and overlapped. For the verification of the results, as a result of checking how many plagiarism suspicious songs are grouped into the same cluster, when the number of clusters was set to 3 in 10 trials, 7.2 pairs of 13 pairs were most often grouped into the same cluster, so K=3 The value was determined by the number of clusters.

Referring to the figure at the bottom of FIG. 6, according to the K-means technique set to K=3, about 96 songs are in the cluster (the first cluster) having the'eyes, nose, and mouth' of the'sun' as the median values. Included. The cluster (the second cluster) with the median value of'Piano Man' of'Mamamoo' contained about 264 sound sources. About 256 sound sources were included in the cluster (third cluster) with the median value of'mackerel' in'Norazo'. The first group had songs with relatively low tempo and little mechanical sound. In the second cluster, rock and K-pop were mixed, and in common there was a strong beat or drum sound and a guitar sound with distortion applied. The third cluster was a cluster of mechanical sound and trot. Despite the mixture of mechanical sound and trot, the distance was relatively short and the density was high compared to other clusters. A list showing 10 songs close to the median value of each cluster is shown at the bottom of FIG. 6.

The GMM module 420 classifies similar music by a Gaussian Mixture Model (GMM) technique. GMM is a technique that is widely used in unsupervised learning in machine learning. Mixture model is a model that assumes that sub-distributions exist in the entire distribution, and that data is generated from multiple distributions with parameters. For example, when one data is captured, the GMM module 420 calculates a probability from the three normal distributions on the left and then allocates the data to the cluster (cluster, group) showing the highest probability.

The graph at the upper right of FIG. 6 is a graph showing BIC values according to the number of clusters in order to find the most suitable K value for this embodiment. In the graph, 3, 4, and 5, which are points where the BIC value sharply bends, were set as initial K values, and clustering (clustering, grouping) was performed by applying the GMM technique. Likewise for GMM, since the median value initially specified is randomly determined, the trend was examined by performing 10 trials. Even after performing clustering 10 times, the songs close to the median were always the same. In addition, the number of clusters was determined by the K=3 value, which was most often grouped into the same cluster with 6.3 pairs out of 13 pairs on average when 10 trials were performed on alleged plagiarism songs. As a result of classifying clusters by setting K=3, the first cluster included approximately 86, the second cluster approximately 293, and the third cluster approximately 237. In addition, when the top 10 songs were selected in the order of the short distance of each cluster, they were in perfect agreement with the results of the Example by the K-means technique.

The DBSCAN module 430 classifies similar music by using a Density-based spatial clustering of applications with noise (DBSCAN) technique. If the K-means technique performs clustering by using the distance between clusters, the DBSCAN module 430 clusters the high-density part because the points are densely concentrated. In simple terms, if there are more than n points within the radius (r) based on a certain point, it is recognized as a cluster, and through the repetition of the task, the core point, the border point that is not the core but belongs to the cluster, does not belong to any place. Clustering is performed with noise. Unlike the K-means technique, there is no need to determine the number of clusters (K).

In the present specification, the data collection module 100, the highlight extraction module 200, the feature extraction module 300, the music classification module 400, the section classification module 210, the highlight determination module 220, the Mel-Filtering module ( 310), MFCC extraction module 320, chroma extraction module 330, tempo calculation module 340, K-means module 410, GMM module 420 and/or DBSCAN module 430 It may be processors that execute the executed processes. Alternatively, it may operate as software modules driven and controlled by a processor. Furthermore, the processor may be a hardware device.

The scope of protection of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the scope of protection of the present invention may not be limited due to obvious changes or substitutions in the technical field to which the present invention pertains.

Data collection module 100, highlight extraction module 200, feature extraction module 300, music classification module 400, section classification module 210, highlight determination module 220, Mel-Filtering module 310, MFCC extraction module 320, chroma extraction module 330, tempo calculation module 340, K-means module 410, GMM module 420, DBSCAN module 430

Claims (5)

A data collection module 100 that collects music of representative singers for each genre from a remote server and removes the live performance version when a digital single version of the same music and a live performance version exist among the collected music;
A highlight extraction module 200 for extracting highlights of the collected music;
A feature extraction module 300 for extracting musical features from the extracted highlights of music; And
A music recommendation system using intrinsic information of music, including a music classification module 400 for classifying collected songs based on the extracted features among similar songs. The method according to claim 1,
The highlight extraction module 200 divides music into an intro, a verse, a chorus, a bridge, and an outro by using a Musical Structural Analysis Framework (MSAF) technique. It includes a section division module 210 for identifying by attaching a label to the section,
The highlight extraction module 200 extracts the chorus and repetition phrases from the music using the label, and then determines the chorus and repetition phrases that are most often repeated in the music among the extracted chorus and repetition phrases, and among the determined chorus and repetition phrases. A music recommendation system using intrinsic music information, comprising: a highlight determination module 220 for determining a section appearing at the beginning of the music as a highlight. The method according to claim 2,
The highlight determination module 220 does not determine the chorus and repeat phrases as highlights when the chorus and repeating phrases determined as highlights exist in front of the 1/4 point of the music, and the next most repeated chorus and repeating phrases A music recommendation system using intrinsic information of music, characterized in that chorus and repetition phrases existing behind a quarter point of are determined as highlights. The method of claim 3,
The feature extraction module 300 includes a Mel-Filtering module 310 that separates and extracts melody data and beat data included in the music highlight, and DCT (Discrete Cosine Transform) processing the audio signal corresponding to the music highlight to MFCC ( An MFCC extraction module 320 that extracts a Mel-Frequency Cepstral Coefficient) coefficient, an audio signal corresponding to a highlight of a music, is processed by STFT (Short-Time Fourier Transform), Constant Q Transform, and CENS (Chroma-Energy Normalized Statistics). A music recommendation system using intrinsic information of music, comprising a chroma extraction module 330 for extracting features and a tempo calculation module 340 for calculating a tempo through bit data separated and extracted from highlights. The method of claim 4,
The music classification module 400 is a K-means module 410 for classifying similar music by a K-means technique based on features extracted by the feature extraction module 300, and similar music by a Gaussian Mixture Model (GMM) technique. Music recommendation using intrinsic information of music, characterized by including a GMM module 420 for classifying music and a DBSCAN module 430 for classifying similar music by a density-based spatial clustering of applications with noise (DBSCAN) technique. system.

Images