KR102771884B1 - Artificial intelligence-based user-customized music recommendation service device - Google Patents

Abstract

The present invention relates to a user-tailored music recommendation service device based on artificial intelligence (AI), and includes: a music database construction unit that analyzes sound sources based on artificial intelligence (AI), segments labeled genres based on BPM, instrument composition, and acoustic characteristics, classifies music emotions, and updates labels to construct a music database; a user situation analysis unit that collects data from a sensor of a user terminal or an external API, generates metadata related to a user's situation through semantic conversion of the collected data, and infers the user's situation; a user emotion analysis unit that infers the user's emotion by linking the user's situation to emotion; a preference music prediction unit that predicts preferred music based on the user's situation and preferred music based on the user's emotion, and determines music to recommend through mapping between the predicted preferred music; and a recommendation music provision unit that provides a recommended music list to the user terminal, selects music selected by the user from the recommended music list from the music database, and provides the music to the user terminal.

Description

{ARTIFICIAL INTELLIGENCE-BASED USER-CUSTOMIZED MUSIC RECOMMENDATION SERVICE DEVICE}

The present invention relates to a music recommendation service technology, and more specifically, to an artificial intelligence (AI)-based customized music recommendation service device that can provide a service of recommending customized music that suits the user's situation and emotions by analyzing the user's situation and musical emotions based on artificial intelligence (AI).

As various smart devices including smartphones are developed, many people enjoy listening to music through streaming (real-time playback) on music platforms. A representative method of listening to music is a playlist. In a survey on actual music streaming usage (2019 Music Industry White Paper, Korea Creative Content Agency), the response rate for ‘listening to edited/saved playlists’ was the highest among ‘15-19 years old (60%)’ and ‘20-24 years old (59.1%)’.

Music recommendation methods can be broadly divided into content-based recommendations and collaborative filtering-based recommendations. Content-based recommendations utilize the unique properties of music, such as the music source including the song and lyrics, to recommend similar types of music that the user listens to. Collaborative filtering-based recommendations analyze the music consumption patterns of users and introduce music that other users with similar tendencies prefer.

However, music is different from other content in that the time spent consuming a single content is short and the consumption tendency changes several times a day depending on emotions and situations, so the existing recommendation method is not suitable. In other words, content-based recommendations and collaborative filtering-based recommendations cause the 'filter bubble' phenomenon, which traps users like a bubble, making it impossible to enjoy new content and diverse cultures.

Recently, prior technologies have been proposed that recommend differentiated music according to the user's situation, but there is a problem that the accuracy of music recommendations that are suitable for the user's situation is low and they do not meet the user's needs because it is difficult to accurately understand the user's current situation and it is still difficult to respond to changes in the situation.

Korean Patent Registration No. 10-1943638 (2019.01.23)

One embodiment of the present invention is to provide an artificial intelligence (AI)-based customized music recommendation service device capable of providing a service that recommends customized music that suits the situation and emotion by analyzing the user's situation and musical emotion based on artificial intelligence (AI).

One embodiment of the present invention provides an artificial intelligence-based user-tailored music recommendation service device that can infer a user's current situation through situation probability inference using sensor data of a mobile device and recommend music suitable for the situation.

One embodiment of the present invention provides an artificial intelligence-based customized music recommendation service device that can infer a user's emotions through pattern analysis of music emotions based on the user's music listening history and recommend music that matches the emotions.

Among the embodiments, the AI-based customized music recommendation service device includes a music database construction unit that analyzes sound sources based on AI, segments labeled genres based on BPM, instrument composition, and acoustic characteristics, classifies music emotions, and updates labels to construct a music database; a user situation analysis unit that collects data from a sensor of a user terminal or an external API, generates metadata related to a user's situation through semantic conversion of the collected data, and infers the user's situation; a user emotion analysis unit that infers the user's emotion by linking the user's situation to emotion; a preference music prediction unit that predicts preferred music based on the user's situation and preferred music based on the user's emotion, and determines music to recommend through mapping between the predicted preferred music; and a recommendation music provision unit that provides a recommended music list to the user terminal, selects music selected by the user from the recommended music list from the music database, and provides the music to the user terminal.

The above music database construction unit may include a step of identifying the spectrum aspect of the sound source, extracting the interval between the initially extracted bit and the final extracted bit through bit extraction processing, and calculating the BPM through interpolation processing; a step of inferring the composition of the instrument used in the sound source through a deep learning instrument classification model; a step of extracting acoustic characteristics of the sound source through a Python spectrum extraction library; and a step of creating a dataset of the BPM, instrument composition, and acoustic characteristics of the sound source and classifying the labeled music genre.

The above musical instrument classification model may be a model trained to extract characteristic values from the spectrum of the signal for each instrument through a Hanning window function, find features that distinguish the sound of the instrument through a Bayesian classification algorithm, and mathematically determine the probability of use of the instrument.

The above music database construction unit extracts data on tempo, dynamics, noise, amplitude change, and brightness of the sound source, and normalizes the values of the average and standard deviation of each extracted data to obtain a probability value of emotion, or extracts the features of the sound source through a two-dimensional CNN music emotion classification model of deep learning, and converts the extracted features into emotion vector values to classify music emotions.

The above user situation analysis unit can preset metadata including user preferred music, activity status, location and time, which serve as criteria for judging the user's situation, and collect raw data required for generating the set metadata from a sensor of the user terminal or an external API.

The above user situation analysis unit can perform situation analysis of the user through situation probability inference using POI (Point of Interest) clustering, naive Bayes, and hierarchical Bayesian network on the collected raw data.

The above user sentiment analysis unit can obtain a Gaussian mixture distribution for the sentiment feature vectors of the music listened to based on the user's recent music listening history, and determine the sentiment with the largest probability value as the user's sentiment.

The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all or only the following effects, and thus the scope of the disclosed technology should not be construed as being limited thereby.

An AI-based customized music recommendation service device according to one embodiment of the present invention can provide a service that recommends customized music that suits the user's situation and emotions by analyzing the user's situation and musical emotions based on artificial intelligence (AI).

An artificial intelligence-based customized music recommendation service device according to one embodiment of the present invention can infer the user's current situation through situation probability inference using sensor data of a mobile device and recommend music suitable for the situation.

An artificial intelligence-based user-tailored music recommendation service device according to one embodiment of the present invention can infer a user's emotions by analyzing patterns of musical emotions based on the user's music listening history and recommend music that matches the emotions.

Figure 1 is a drawing explaining a music recommendation service system according to the present invention.
Fig. 2 is a drawing explaining the system configuration of the music recommendation service device of Fig. 1.
Fig. 3 is a drawing explaining the functional configuration of the music recommendation service device of Fig. 1.
Figure 4 is a flowchart illustrating a method for providing a user-tailored music recommendation service based on artificial intelligence according to the present invention.
FIG. 5 is a drawing explaining one embodiment of a sound source characteristic-based emotion classification method according to the present invention.
FIG. 6 is a drawing explaining another embodiment of a sound source characteristic-based emotion classification method according to the present invention.

The description of the present invention is only an embodiment for structural and functional explanation, so the scope of the rights of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiments can be variously modified and can have various forms, the scope of the rights of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the purpose or effect presented in the present invention does not mean that a specific embodiment must include all of them or only such effects, so the scope of the rights of the present invention should not be understood as being limited thereby.

Meanwhile, the meanings of the terms described in this application should be understood as follows.

Terms such as "first", "second", etc. are intended to distinguish one component from another, and the scope of the rights should not be limited by these terms. For example, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

When it is said that a component is "connected" to another component, it should be understood that it may be directly connected to that other component, but there may also be other components in between. On the other hand, when it is said that a component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

A singular expression should be understood to include the plural expression unless the context clearly indicates otherwise, and terms such as "comprises" or "have" should be understood to specify the presence of a feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In each step, the identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of each step, and each step may occur in a different order than stated unless the context clearly indicates a specific order. That is, each step may occur in the same order as stated, may be performed substantially simultaneously, or may be performed in the opposite order.

The present invention can be implemented as a computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all kinds of recording devices that store data that can be read by a computer system. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. In addition, the computer-readable recording medium can be distributed over network-connected computer systems, so that the computer-readable code can be stored and executed in a distributed manner.

All terms used herein, unless otherwise defined, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the contextual meaning of the relevant art, and shall not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

Figure 1 is a drawing explaining a music recommendation service system according to the present invention.

Referring to FIG. 1, a music recommendation service system (100) can be implemented including a user terminal (110), a music recommendation service device (130), and a database (150).

The user terminal (110) is a device capable of playing music, and may correspond to a computing device capable of using an artificial intelligence-based customized music recommendation service, and may be implemented as a smart phone, a laptop, or a tablet PC, and is not necessarily limited thereto, and may also be implemented as various mobile devices including a connected car, etc. Here, the user terminal (110) may include various sensors such as a motion sensor, a position sensor, and an environment sensor to provide raw data necessary for analyzing the user's situation. The user terminal (110) may be connected to a music recommendation service device (130) through a network, and a plurality of user terminals (1100) may be connected to the music recommendation service device (130) simultaneously.

The music recommendation service device (130) may be implemented as a server corresponding to a computer or program that collects user context information through various sensor information of the user terminal (110) to understand the user's current situation (Context Awareness), infers emotions based on the user's situation and listening history, and recommends music that matches the situation and emotions. The music recommendation service device (130) may be connected to the user terminal (110) through a wired network or a wireless network such as Bluetooth, WiFi, LTE, etc., and may transmit and receive data with the user terminal (110) through the network.

In one embodiment, the music recommendation service device (130) may store data necessary for providing an artificial intelligence-based customized music recommendation service in conjunction with a database (150). Meanwhile, unlike FIG. 1, the music recommendation service device (130) may be implemented by including the database (150) therein. In addition, the music recommendation service device (130) may be implemented by including an API module for providing a music recommendation service. More specifically, the music recommendation service device (130) may provide an API interface that can access information about a user profile, a music profile, and recommended music to an external system, and the external system may independently access various data managed by the music recommendation service device (130) through the API interface.

The database (150) may correspond to a storage device that stores various information required in the operation process of the music recommendation service device (130). The database (150) may store information on the user's music listening history, and may store information for situation probability inference using sensor data of the user terminal (110). However, the database is not necessarily limited thereto, and may store information collected or processed in various forms in the process of the music recommendation service device (130) providing a user-tailored music recommendation service based on artificial intelligence.

Fig. 2 is a drawing explaining the system configuration of the music recommendation service device of Fig. 1.

Referring to FIG. 2, the music recommendation service device (130) may include a processor (210), a memory (230), a user input/output unit (250), and a network input/output unit (270).

The processor (210) can execute a procedure for processing each step in the process in which the music recommendation service device (130) operates, manage the memory (230) that is read or written throughout the process, and schedule a synchronization time between the volatile memory and the non-volatile memory in the memory (230). The processor (210) can control the overall operation of the music recommendation service device (130), and is electrically connected to the memory (230), the user input/output unit (250), and the network input/output unit (270) to control the data flow therebetween. The processor (210) can be implemented as a CPU (Central Processing Unit) of the music recommendation service device (130).

The memory (230) may include an auxiliary memory device implemented as a non-volatile memory such as an SSD (Solid State Disk) or an HDD (Hard Disk Drive) and used to store all data required for the music recommendation service device (130), and may include a main memory device implemented as a volatile memory such as a RAM (Random Access Memory).

The user input/output unit (250) may include an environment for receiving user input and an environment for outputting specific information to the user. For example, the user input/output unit (250) may include an input device including an adapter such as a touch pad, a touch screen, a visual keyboard, or a pointing device, and an output device including an adapter such as a monitor or a touch screen. In one embodiment, the user input/output unit (250) may correspond to a computing device connected via a remote connection, and in such a case, the music recommendation service device (130) may be performed as an independent server.

The network input/output unit (270) provides a communication environment for connection to a user terminal (110) via a network, and may include an adapter for communication such as, for example, a LAN (Local Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), and a VAN (Value Added Network).

Fig. 3 is a drawing explaining the functional configuration of the music recommendation service device of Fig. 1.

Referring to FIG. 3, the music recommendation service device (130) may include a music database construction unit (310), a user situation analysis unit (330), a user emotion analysis unit (350), a preferred music prediction unit (370), a recommended music provision unit (390), and a control unit (not shown in FIG. 3).

The music database construction unit (310) can analyze sound sources based on artificial intelligence (AI) to construct a music database. Currently, in the case of large-scale domestic sound source providers, only simple meta information such as album name, sound source title, singer, release date, and simple genre are often labeled, which limits music recommendation. The music database construction unit (310) can analyze sound source characteristics and classify genres and emotions based on the analyzed sound source characteristics through a deep learning model to construct a music database. In one embodiment, the music database construction unit (310) can analyze BPM (Beats Per Minute), instrument composition, and other acoustic characteristics of the sound source and classify genres through a deep learning model. Here, BPM is a unit that indicates tempo in music. Audio files can be converted into spectra. Therefore, the music database construction unit (310) can identify the spectrum aspect of the sound source, extract the interval between the initially extracted bit and the finally extracted bit through bit extraction processing, and automatically calculate BMP through interpolation processing.

In addition, the music database construction unit (310) can infer the instrument composition of the sound source through the deep learning model. In the sound source, the instrument composition can be an indicator that specifies the mood of the sound source and distinguishes the detailed genre. Here, the instrument classification model using deep learning is a model that has been trained to extract characteristic values through the spectrum of the signal of each instrument through the Hanning window function, find features that distinguish the sound of the instrument through the Bayes' classification algorithm, and mathematically determine which instrument was used in the sound source. The music database construction unit (310) can infer the instrument composition of the sound source by obtaining data in which the presence or absence (probability) of the instrument used in the sound source is expressed as a real number from 0 to 1 through the deep learning instrument classification model.

In addition, the sound source database construction unit (310) can extract acoustic features such as MFCC (Mel Frequency Cepstral Coefficient), Mel-spectrogram, centroid, and chromagram using Python's spectrum extraction library for the sound source. The sound source database construction unit (310) can create a data set of BPM, instrument composition, and acoustic characteristics of the sound source, and perform deep learning model training using the created data set and existing genre classification data as labels to classify the music genre in more detail than before.

In addition, the sound source database construction unit (310) can classify music emotions based on sound source characteristics. In one embodiment, the sound source database construction unit (310) can extract data on tempo, dynamics, noise, amplitude change, and brightness of the sound source, and can obtain each emotion as a probability by normalizing the average and standard deviation values of each extracted data. In another embodiment, the sound source database construction unit (310) can extract the features of the sound source through a two-dimensional CNN model of deep learning, and convert the extracted features into an emotion vector to classify the emotions expressed by the sound source.

In addition, the sound source database construction unit (310) can analyze the user's situation and emotion on SNS and apply it to music classification. The sound source database construction unit (310) can identify the user's situation or emotion regarding music by collecting and analyzing the situation, emotion, and style tag data disclosed by the site providing newsfeed, social data, and public API. In one embodiment, the sound source database construction unit (310) can collect social media mention data, label it through machine learning, analyze the morphology of the data mentioning the sound source and the singer, assign a score according to the similarity between sentences and keywords, synonyms, and context, and sort and filter the data through Bayesian network learning to extract and analyze the frequency. At this time, since the reliability of the data collected from social media can vary depending on the number of mentions and the influence of the posting site, a social media mention collection data model can be constructed by considering how much the site was cited, the influence of the subscribers of the site (article), and the number of subscribers. This can be defined as in the following mathematical expression 1.

[Mathematical formula 1]

Here, is the total number of mentions (citations) to a particular site, is the number of link data where the sound source is mentioned, is the number of traffic from a specific site citation (site), is the number of traffic to a specific site.

Table 1 below shows an example of a social media mention collection data model.

[Table 1]

The user situation analysis unit (330) can collect data from a sensor of the user terminal (110) or an external API to analyze the user's situation. In one embodiment, the user situation analysis unit (330) can preset metadata that serve as a basis for judging the user's situation and collect data necessary to generate the set metadata. Here, the metadata may include user preference, activity status, location, time, etc. The user preference is the type of music that the user prefers. The user activity status may include being in a means of transportation such as a car (IN_VEHICLE), being on a bicycle (ON_BICYLE), walking or running (ON_FOOT), running (RUNNING), standing (STILL), gravity change such as going up to a high place or going down to a low place (TILTING), walking (WALKING), unknown (UNKNOWN), and the like, and may include status changes such as starting (Start), stopping (Stop), and being active (During) for a specific activity. Places can be divided into user-centric important places and service-centric special places, and can include access frequency, time, location, and type of the place, and can include status changes such as exiting the place, entering the place, and already being in the place (In). Time can include the time when the user performs a specific action, day of the week, every day, a specific time, or near a specific time, and can include entering (IN) within a set time zone as a status change.

The user situation analysis unit (330) can collect raw data that can generate set metadata from sensors of the user terminal (110) or an external API. The user terminal (110) can have numerous sensors built in, and the user terminal (110) can measure the operation, location, and surrounding environment of the user terminal (110) through the built-in sensors. For example, the user terminal (110) can have an acceleration sensor, a light sensor, a GPS sensor, a proximity sensor, a microphone, and an acoustic sensor. In one embodiment, the user situation analysis unit (330) can collect only the data necessary for generating the set metadata among the sensors in the user terminal (110), and collect data only when a change occurs in the collected data compared to previously collected data, thereby reducing communication costs and reducing the weight of the data capacity to be processed.

The user situation analysis unit (330) can perform user situation analysis by using POI (Point of Interest) clustering, Naive Bayes, and situation probability inference using a hierarchical Bayesian network for the sensor data of the user terminal (110). For example, the user situation analysis unit (330) can determine the user's current situation as a way to exercise on Wednesday afternoon by judging the main movement location (home, office, etc.), weather, movement speed (exercise, drive, etc.), season, and presence of exceptional circumstances through learning GPS sensor information, WiFi (SSID), and Bluetooth sensor values.

The user emotion analysis unit (350) can infer the user's emotion by connecting the user's situation with the emotion. In one embodiment, the user emotion analysis unit (350) can infer the user's current emotional state by analyzing the user's activity state. Here, the user emotion analysis unit (350) can analyze the user's emotion by using a pre-learned emotion analysis model that uses at least two of the user's activity state, location, and time generated by the user situation analysis unit (330) as input data.

The user emotion analysis unit (350) can also infer the user's emotion by analyzing the pattern of the user's recently listened music emotion. In one embodiment, the user emotion analysis unit (350) can obtain a Gaussian mixture density for the emotion feature vector of the user's listened music, and determine the emotion of the GMM with the largest probability value as the user's emotion.

The preferred music prediction unit (370) can predict preferred music based on the user's situation and preferred music based on the user's emotion, respectively, and determine music to recommend through mapping between the predicted preferred music. In one embodiment, the preferred music prediction unit (370) can select N pieces of music (where N is a natural number) of genres with sound source characteristics and situation tags matching the user's situation from among the music stored in the music database constructed in the music database construction unit (310) to generate a first preferred music list. The preferred music prediction unit (370) can select N pieces of music classified with music emotions matching the user's emotions from among the music stored in the music database to generate a second preferred music list. The preferred music prediction unit (370) can determine a recommended music list by merging the first preferred music list and the second preferred music list.

The recommended music providing unit (390) can provide a recommended music list to the user terminal (110) and select music selected by the user from the recommended music list from the music database and provide it to the user terminal (110). In one embodiment, the recommended music providing unit (390) can update the recommended music list according to the change in the user's situation or emotion while the user is listening to the recommended music and provide it to the user terminal (110).

The control unit (not shown in FIG. 3) controls the overall operation of the music recommendation service device (130) and can manage the control flow or data flow between the music database construction unit (310), the user situation analysis unit (330), the user emotion analysis unit (350), the preferred music prediction unit (370), and the recommended music provision unit (390).

Figure 4 is a flowchart illustrating a method for providing a user-tailored music recommendation service based on artificial intelligence according to the present invention.

Referring to FIG. 4, the music recommendation service device (130) can analyze sound sources based on artificial intelligence through the music database construction unit (310) to construct a music database (step S410). The music database construction unit (310) can classify sound sources into detailed genres and emotions through a deep learning model and label situation, emotion, and style tag data through analysis of sound source mention data collected from social media such as Facebook, Twitter, community comments, blogs, and news to construct a music database. Accordingly, detailed music recommendations that meet the user's needs can be made.

In addition, the music recommendation service device (130) can analyze the user's current situation through situation probability inference by collecting sensor data of the user terminal (110) through the user situation analysis unit (330) (step S430). The music recommendation service device (130) can analyze the user's emotions through pattern analysis of the user's current situation or recently listened music emotions through the user emotion analysis unit (350) (step S450).

The music recommendation service device (130) can predict a first preferred music list according to the user's situation and a second preferred music list according to the user's emotions through the preferred music prediction unit (370), and then merge them to determine a recommended music list (step S470). The music recommendation service device (130) provides a recommended music list customized to the user's situation and emotions to the user terminal (110) through the recommended music provision unit (390) (step S490).

FIG. 5 is a drawing explaining one embodiment of a sound source characteristic-based emotion classification method according to the present invention.

Referring to Fig. 5, about 2,000 uniform data can be prepared using 12 labels related to various emotions such as excited, happy, pleased, annoyed, and angry. Then, data on tempo, dynamics, noise, amplitude change, and brightness of the sound source included in the dataset can be extracted using the Python library, and the average and standard deviation values of each extracted data can be used to normalize the distribution. The average and standard deviation values of tempo, dynamics, noise, amplitude change, and brightness of the new sample sound source can be substituted into the previously obtained normal distribution curve of each part to derive the probability value. The X (Arousal), Y (Valence) coordinate values can be derived by multiplying each probability by the weights assigned to the X-axis and Y-axis, and the area occupied by the plane divided into 12 in advance by drawing a circle centered on the obtained X and Y coordinate values can be derived to derive the probability of each emotion.

FIG. 6 is a drawing explaining another embodiment of a sound source characteristic-based emotion classification method according to the present invention.

Referring to Fig. 6, after converting an audio stereo file into a mono file, the audio data is decomposed into frequency components through Fourier transform, and a spectrogram conversion process is performed with a small resolution to extract two-dimensional data that can most similarly represent what the human ear hears. After preprocessing the two-dimensional data in the form of a spectrogram, it is applied to a music emotion classification model to extract features, and the extracted features are converted into an emotion vector to classify the emotion expressed by the sound source. Here, the music emotion classification model is a two-dimensional CNN model of deep learning.

The present invention can increase the accuracy of emotion classification through a combination of the probability value of emotion extracted in FIG. 5 and the emotion vector value extracted in FIG. 6.

An AI-based personalized music recommendation service device according to one embodiment can recommend a personalized music playlist for a user by predicting the user's situation and emotions, thereby satisfying the user's needs. Furthermore, it can introduce various types of music to consumers in a personalized manner, not limited to specific popular music, and provide creators with an opportunity to promote their creative activities.

Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

100: AI-based personalized music recommendation service system
110: User terminal 130: Music recommendation service device
150: Database
210: Processor 230: Memory
250: User I/O section 270: Network I/O section
310: Music database construction department 330: User situation analysis department
350: User sentiment analysis section 370: Preferred music prediction section
390: Recommended Music Provider

Claims (7)

A music database construction department that analyzes sound sources based on artificial intelligence (AI), segments the labeled genres based on BPM, instrument composition, and acoustic characteristics, and classifies music emotions to update the labels to build a music database;
A user situation analysis unit that presets metadata including user-preferred music, activity status, location, and time that serve as criteria for judging the user's situation, collects raw data required for generating the set metadata from a sensor on the user's terminal or an external API, and generates metadata related to the user's situation through meaning conversion of the collected data to infer the user's situation;
A user sentiment analysis unit that infers the sentiment of the user by using a pre-learned sentiment analysis model that uses at least two of activity status, location, and time as input data in the user's situation;
A preference music prediction unit that predicts preferred music based on the user's situation and preferred music based on the user's emotion, and determines music to recommend through mapping between the predicted preferred music; and
Including a recommended music providing unit that provides a recommended music list to the user terminal and selects music selected by the user from the recommended music list from the music database and provides the music to the user terminal,
The above music database construction department
A step of identifying the spectral aspect of the sound source, extracting the interval between the initially extracted bit and the final extracted bit through bit extraction processing, calculating the BPM through interpolation processing, inferring the composition of the instrument used in the sound source through a deep learning instrument classification model, extracting acoustic characteristics of the sound source through Python's spectrum extraction library, and creating a dataset of the BPM, instrument composition, and acoustic characteristics of the sound source to classify and segment the labeled music genre; and
i) extracting data on tempo, noise, and amplitude changes of the sound source, and obtaining emotion as a probability value by normalizing the values of the average and standard deviation of each extracted data, extracting the features of the sound source through a two-dimensional CNN music emotion classification model of deep learning, converting the extracted features into emotion vector values, and classifying music emotions through a combination of the probability values of the emotions and the emotion vector values, or ii) collecting social media mention data, labeling it through machine learning, analyzing the morphology of data mentioning the sound source and the singer, assigning scores according to the similarity between sentences and keywords, synonyms, and context, and analyzing the user's situation and emotion regarding music through Bayesian network learning, and applying it to music classification.
The above user situation analysis section
Collect data only when changes have occurred compared to previously collected data.
The above preferred music prediction section
An artificial intelligence-based user-tailored music recommendation service device characterized in that a first preferred music list is created by selecting N pieces of music (where N is a natural number) of a genre having sound source characteristics and context tags matching the user's situation from among the music stored in the music database constructed in the music database construction unit, and a second preferred music list is created by selecting N pieces of music classified by music emotions matching the user's emotions from among the music stored in the music database, and a recommended music list is determined by merging the first preferred music list and the second preferred music list.
delete In the first paragraph, the instrument classification model
An artificial intelligence-based customized music recommendation service device characterized by a model trained to extract characteristic values through the spectrum of the signal for each instrument through the Hanning window function, find features that distinguish the sound of the instrument through the Bayesian classification algorithm, and mathematically determine the probability of use of the instrument.
delete delete In the first paragraph, the user situation analysis unit
An artificial intelligence-based customized music recommendation service device characterized in that it performs situation analysis of the user through situation probability inference using POI (Point of Interest) clustering, naive Bayes, and hierarchical Bayesian network on the collected raw data.
In the first paragraph, the user emotion analysis unit
An artificial intelligence-based personalized music recommendation service device characterized by calculating a Gaussian mixture distribution of emotional feature vectors of music listened to based on the user's recent music listening history and determining the emotion with the highest probability value as the user's emotion.