KR102907945B1 - Artificial intelligence based mididata generating solution providing system - Google Patents

Abstract

A system for providing an AI-based MIDI data extraction solution is provided, and includes an extraction service providing server including a user terminal for selecting an original music file in order to generate MIDI data of a MIDI file to be input into a karaoke accompaniment machine, a song analysis unit for inputting the original music file into a pre-built AI to convert it into a MIDI file, and then extracting tempo, key, composition, song structure, range, and genre as feature data from the MIDI file, a setting unit for measuring the similarity between a vector embedding MIDI data of a pre-built MIDI data database and a vector embedding feature data, and then extracting similar MIDI data based on the similarity, and a setting unit for setting the pattern of values of instruments, equalizers, and filters applied to the similar MIDI data as candidate MIDI data, and a generation unit for generating MIDI data for the original music file by performing a pre-set sound optimization process on the candidate MIDI data.

Description

AI-based MIDI data extraction solution provision system {ARTIFICIAL INTELLIGENCE BASED MIDIDATA GENERATING SOLUTION PROVIDING SYSTEM}

The present invention relates to a system for providing an AI-based MIDI data extraction solution, and provides a method for automatically extracting MIDI data from an original music file.

MIDI (Music Instrument Digital Interface) is a digital protocol for exchanging musical data between computers and musical instruments. Unlike MP3s or WAVs, which store sounds directly, MIDI stores performance data, such as which notes to play, when, and how loudly. The MIDI input into karaoke accompaniment devices isn't MR (Music Recorded) files like MP3s or WAVs, which are the same as what actual singers sing, but rather MIDI-standard data (files). The MR files used by singers can be difficult for the average person to sing, as the vocal melody line is not prominent, and the beginning part can be confusing. Furthermore, key and speed adjustments are not possible. Therefore, MIDI-standard data (files) are still used in karaoke accompaniment devices. To create these MIDI files, music majors listen to the original music files, transcribe the scores for each instrument into a dedicated program, and mix the accompaniment.

At this time, a method of generating or reproducing accompaniment data was studied and developed. In relation to this, prior art Korean Patent No. 10-2290901 (announced on August 19, 2021) and Korean Patent No. 10-1881854 (announced on July 25, 2018) disclose a configuration in which, when a song is requested from a user terminal, TTS (Text-to-Speech) data to be output between the lyrics lines is generated based on a lyrics format including time information for each lyric line of the song and time information for each syllable of the lyrics, TTS data is generated as accompaniment data together with MR data and transmitted to the user terminal, and the user terminal executes the accompaniment data, and a configuration in which MIDI data and lyrics data are extracted and then played so that MIDI can be played without using a karaoke accompaniment machine, respectively.

However, in the former case, it does not create an MR, which is an accompaniment sound source, but only discloses the configuration after the MR data has already been created. In the latter case, it only discloses the configuration that allows the MR data to be output on devices other than a karaoke accompaniment machine after it already exists. As mentioned above, in order to create an MR, the original music file is actually listened to by ear by music majors and is manually produced one by one, which consumes excessive time and money, and even then, the quality of the results varies widely. In other words, each MR inevitably reflects the music major's personal background knowledge, experience, taste, and opinion. For example, during the arrangement process, a long intro may be simplified, a rubato part may be deleted, and a fade-out or fermata of a difficult performance may be deleted. Therefore, research and development of a system that can convert original music files into MIDI data based on AI is required.

One embodiment of the present invention provides an AI-based MIDI data extraction solution system that converts an original music file into a MIDI file using AI to minimize manual work so that variations do not occur for each individual, and facilitates the supply and demand of MIDI through an automated process, analyzes song information by extracting tempo, key/key, song structure, range, and genre from the MIDI file, sets the pattern of the most similar MIDI data as candidate MIDI data through a similarity comparison with MIDI data stored in a pre-built MIDI data database, and performs a preset sound optimization process on the candidate MIDI data, thereby automatically generating MIDI data for the original music file. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem described above, and other technical problems may exist.

As a technical means for achieving the above-described technical task, one embodiment of the present invention includes an extraction service providing server including a user terminal for selecting an original music file to generate MIDI data of a MIDI file to be input into a karaoke accompaniment machine, a song analysis unit for inputting the original music file into a pre-built AI to convert it into a MIDI file, and then extracting tempo, key, composition, song structure, range, and genre as feature data from the MIDI file, a setting unit for measuring the similarity between a vector embedding MIDI data of a pre-built MIDI data database and a vector embedding feature data, and then extracting similar MIDI data based on the similarity, and setting a pattern of values of an instrument, an equalizer, and a filter applied to the similar MIDI data as candidate MIDI data, and a generation unit for generating MIDI data for the original music file by performing a pre-set sound optimization process on the candidate MIDI data.

According to any one of the problem solving means of the present invention described above, in order to prevent individual deviations from occurring, minimize manual work, and facilitate the supply and demand of MIDI through an automated process, the original music file is converted into a MIDI file using AI, the song information is analyzed by extracting tempo, key/key, song structure, range, and genre from the MIDI file, and the pattern of the most similar similar MIDI data is set as candidate MIDI data through a similarity comparison with the MIDI data stored in a pre-built MIDI data database, and a preset sound optimization process is performed on the candidate MIDI data, thereby automatically generating MIDI data for the original music file, thereby preventing individual deviations from occurring, preventing a lack of core assets of karaoke rooms through a sufficient supply of MIDI, and maximizing profit generation.

FIG. 1 is a diagram for explaining a system for providing an AI-based media data extraction solution according to one embodiment of the present invention.
Figure 2 is a block diagram illustrating an extraction service providing server included in the system of Figure 1.
FIG. 3 and FIG. 4 are drawings for explaining an embodiment in which an AI-based media data extraction service according to one embodiment of the present invention is implemented.
FIG. 5 is a flowchart illustrating a method for providing an AI-based media data extraction service according to one embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity of description, and similar reference numerals have been used throughout the specification to indicate similar elements.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Furthermore, when a part is said to "include" a component, this should be understood to mean that, unless specifically stated to the contrary, it may include other components rather than excluding them, and does not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

The terms "about," "substantially," and the like used throughout the specification are used in a sense of degree or in a sense close to the numerical value when manufacturing and material tolerances inherent to the meanings mentioned are presented, and are used to prevent unscrupulous infringers from unfairly exploiting disclosures that mention precise or absolute numerical values to aid understanding of the present invention. The terms "step of doing" or "step of" used throughout the specification of the present invention do not mean "step for doing."

In this specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be realized by using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, the "unit" is not limited to software or hardware, and the "unit" may be configured to be on an addressable storage medium or may be configured to reproduce one or more processors. Accordingly, as an example, the "unit" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and "units" may be combined into a smaller number of components and "units," or further separated into additional components and "units." Additionally, components and '~parts' may be implemented to regenerate one or more CPUs within a device or secure multimedia card.

Some of the operations or functions described herein as being performed by a terminal, apparatus, or device may instead be performed by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described herein as being performed by a server may also be performed by a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as terminal and mapping or matching may be interpreted to mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data.

The present invention will be described in detail with reference to the attached drawings below.

FIG. 1 is a diagram illustrating an AI-based midi data extraction solution providing system according to one embodiment of the present invention. Referring to FIG. 1, the AI-based midi data extraction solution providing system (1) may include at least one user terminal (100), an extraction service providing server (300), and at least one karaoke accompaniment machine (400). However, since the AI-based midi data extraction solution providing system (1) of FIG. 1 is merely one embodiment of the present invention, the present invention is not limited to FIG. 1.

At this time, each component of FIG. 1 is generally connected via a network (Network, 200). For example, as illustrated in FIG. 1, at least one user terminal (100) may be connected to an extraction service providing server (300) via a network (200). In addition, the extraction service providing server (300) may be connected to at least one user terminal (100) and at least one karaoke accompaniment machine (400) via the network (200). In addition, at least one karaoke accompaniment machine (400) may be connected to an extraction service providing server (300) via the network (200).

Here, a network means a connection structure that enables information exchange between each node, such as multiple terminals and servers, and examples of such networks include a local area network (LAN), a wide area network (WAN), the Internet (WWW), a wired and wireless data communication network, a telephone network, and a wired and wireless television communication network. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), 5G NR (New Radio), 6G (6th Generation of Cellular Networks), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth networks, NFC (Near-Field Communication) networks, satellite broadcasting networks, analog broadcasting networks, and DMB (Digital Multimedia Broadcasting) networks.

In the following, the term "at least one" is defined as a term including both singular and plural, and it will be clear that even if the term "at least one" does not exist, each component can exist in the singular or plural and can mean either the singular or plural. Furthermore, whether each component is provided in the singular or plural may vary depending on the embodiment.

At least one user terminal (100) may be a terminal of a person in charge or manager who inputs an original music file using a web page, app page, program or application related to an AI-based MIDI data extraction service and edits or manages the quality of the extracted MIDI data.

Here, at least one user terminal (100) may be implemented as a computer capable of accessing a remote server or terminal via a network. Here, the computer may include, for example, a notebook, desktop, or laptop equipped with a navigation system or web browser. In this case, at least one user terminal (100) may be implemented as a terminal capable of accessing a remote server or terminal via a network. At least one user terminal (100) may include, for example, a wireless communication device that ensures portability and mobility, and may include all types of handheld-based wireless communication devices such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, etc.

The extraction service providing server (300) may be a server providing an AI-based MIDI data extraction service web page, app page, program, or application. Furthermore, the extraction service providing server (300) may be a server that, when an original music file is input into a user terminal (100), generates MIDI data for the original music file and then provides the data to the user terminal (100). Furthermore, when editing is performed on the user terminal (100), the server may be a server that stores the edited MIDI data and then generates a MIDI file to be played on a karaoke accompaniment machine (400).

Here, the extraction service providing server (300) may be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a notebook computer, desktop computer, or laptop computer equipped with a navigation system or web browser.

At least one karaoke accompaniment device (400) may be a device that receives and plays MIDI files using a web page, app page, program, or application related to an AI-based MIDI data extraction service. The karaoke accompaniment device (400) may be a karaoke accompaniment device located in a karaoke room, as shown in FIG. 1, but may also be a general smartphone device that outputs MIDI files.

Here, at least one karaoke accompaniment device (400) may be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a notebook, desktop, or laptop equipped with a navigation system or web browser. In this case, at least one karaoke accompaniment device (400) may be implemented as a terminal capable of connecting to a remote server or terminal via a network. At least one karaoke accompaniment device (400) may include, for example, all kinds of handheld-based wireless communication devices such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, etc., as wireless communication devices that ensure portability and mobility.

FIG. 2 is a block diagram for explaining an extraction service providing server included in the system of FIG. 1, and FIGS. 3 and 4 are drawings for explaining an embodiment in which an AI-based media data extraction service according to an embodiment of the present invention is implemented.

Referring to FIG. 2, the extraction service providing server (300) may include a song analysis unit (310), a setting unit (320), a generation unit (330), and an accompaniment playback unit (340).

When the extraction service providing server (300) according to one embodiment of the present invention or another server (not shown) that operates in conjunction with it transmits an AI-based midi data extraction service application, program, app page, web page, etc. to at least one user terminal (100) and at least one karaoke accompaniment machine (400), the at least one user terminal (100) and the at least one karaoke accompaniment machine (400) may install or open the AI-based midi data extraction service application, program, app page, web page, etc. In addition, the service program may be driven in the at least one user terminal (100) and the at least one karaoke accompaniment machine (400) using a script executed in a web browser. Here, a web browser is a program that enables the use of a web (WWW: World Wide Web) service and means a program that receives and displays hypertext described in HTML (Hyper Text Mark-up Language), and includes, for example, Chrome, Edge (Microsoft Edge), Safari, Firefox, Whale, UC Browser, etc. Additionally, the application refers to an application on a terminal, and includes, for example, an app running on a mobile terminal (smartphone).

<Song Information Analysis>

<Midi file conversion>

Referring to FIG. 2, the song analysis unit (310) can input an original music file into a pre-built AI, convert it into a MIDI file, and then extract tempo, key, composition, song structure, range, and genre as feature data from the MIDI file. The user terminal (100) can select the original music file to generate MIDI data of the MIDI file to be input into the karaoke accompaniment machine.

At this time, the built-in AI can be an automatic conversion program that analyzes the original music file and converts it into a MIDI file. For example, you can use AnthemScore, an AI-based automatic conversion program, Melodyne, an advanced audio pitch analysis program, Sonic Visualiser, which converts to MIDI after spectrum analysis, AmazingMIDI for melody extraction, etc., but are not limited to these. Alternatively, you can convert it in a DAW (Digital Audio Workstation). You can convert the original music file into a MIDI file using a DAW such as FL Studio, Ableton Live, Cubase, or Logic Pro. At this time, Ableton Live provides the Convert Melody to MIDI function, FL Studio can convert audio into a MIDI file with the NewTone plug-in, and Logic Pro X can use the Flex Pitch function. Alternatively, you can simply convert audio into a MIDI file on a website. For example, you can use Bear Audio, Evano, or AudioToMidi. Alternatively, you can do the conversion directly through coding using Python + Librosa library. You can also use Librosa library in Python to analyze audio and convert it to MIDI file.

At this time, the best method is to convert the original music file into a MIDI file using the above-described automation program and then perform additional editing in a DAW. Accordingly, in one embodiment of the present invention, after converting the original music file into a MIDI file, the information that constitutes the song is analyzed through song analysis, similar MIDI data is extracted, and the pattern of the similar MIDI data is set as candidate MIDI data, and here, MIDI data for the original music file is generated through a sound optimization process.

<Song Analysis>

If the original music file is converted into a MIDI file, the song analysis unit (310) can start song analysis to extract ① tempo, ② key, tonality, ③ song structure, ④ range, and ⑤ genre. To this end, ① the song analysis unit (310) can analyze the tempo by extracting the tempo meta event in the MIDI file when analyzing the tempo. At this time, the tempo meta event is an event that adjusts the tempo (speed) of the song in the MIDI file. The MIDI file only stores performance information, but the tempo meta event sets the BPM (Beat Per Minute) of the song to adjust the playback speed.

Tempo Meta Event BPM (beat rate) setting Determine the overall tempo of the song from the tracks in the MIDI file Tempo can be changed mid-song Can be converted from 120BPM to 90BPM Editing tool Modifiable in MIDI sequencer or DAW Code format FF 51 03 tt tt tt

That is, you can freely adjust the tempo in a MIDI file by using the tempo meta event. Here, FF 51 is the tempo meta event, 03 is the data length (3 bytes), and tt tt tt means 1/4 beat (Quarter Note) time in microseconds. For example, in the case of 120 BPM, it can be expressed as FF 51 03 07 A1 20, so 07 A1 20 (hexadecimal) = 500,000 microseconds (0.5 seconds) = 120 BPM. The tempo (BPM) formula is 60000000 divided by the 1/4 beat length in microseconds.

② The song analysis unit (310) can analyze the key and scale by identifying the most frequent range and chord progression pattern through the note distribution of the MIDI file. First, all note data can be extracted from the MIDI file, and then the number and frequency of each note can be analyzed. Notes are classified into C, C#, D, D#...B (12 notes in total), and it is possible to check which note is used most frequently in a given song. For example, if the notes C, E, and G are used frequently, this may be the key of C Major (C Major), and if the notes A, C, and E appear mainly, this may be the key of A Minor (A Minor). When analyzing the frequency of notes, note frequency analysis (Pitch Class Distribution) can be used, and the key and tonality can be analyzed using the similarity score with each scale (Scale Matching Score). In summary, by analyzing the note distribution, you can find the most frequently appearing note (most frequent note range), and by comparing the major and minor scales, you can find the optimal key (scale).

③ The song analysis unit (310) can analyze the sections corresponding to the intro, verse, chorus, interlude, and ending in the MIDI file using a pre-built song structure analysis model. At this time, the song structure analysis model can be visually analyzed in a DAW (FL Studio, Ableton, Cubase, etc.), or the song structure can be identified by performing waveform analysis using a program such as Melodyne or Sonic Visualizer, or it can be automatically analyzed using Python.

Components explanation Intro The beginning of the song sets the mood Verse The part where the story (lyrics) unfolds Pre-Chorus Gradual rise before moving on to the chorus Chorus The most intense and memorable part (chorus) Bridge A part that changes the flow of the song Outro (Ending) The part that ends the song

If we analyze the structure of modern popular music, it generally progresses in the form of Intro, Verse 1, Pre-Chorus (Build-up), Chorus (Drop), Verse 2, Pre-Chorus (Build-up), Chorus (Drop), Bridge, Chorus (Drop), and Outro. Intro and Outro refer to the measures that express the beginning and end of the song. Verse means [section], and in the introduction-development-turn-conclusion (起承轉結) structure, it refers to the [起] part. Pre-chorus serves to connect the chorus that follows and refers to the [承] in the introduction-development-turn-conclusion (起承轉結) structure. Chorus expresses the most dramatic part of the song and refers to the [轉] in the introduction-development-turn-conclusion (起承轉結) structure. The interlude typically expresses a slightly different mood from the preceding verse, pre-chorus, and chorus, but it serves as a natural bridge that connects them. Contemporary popular music with this format typically repeats the same chords, excluding the bridge section, and uses changes in instrumentation or rhythmic feel around bars 8 or 16 to create a sense of development within the overall feel of the song. Of these, the chorus, the refrain, represents the most dramatic part of the song, making it the most splendid and richly expressed sonically. It is filled with instruments spanning from low to high frequencies.

import mido
from collections import Counter

# Importing MIDI files
midi_file = mido.MidiFile('example.mid')

# Hourly note analysis
note_counts = Counter()

# Extracting time patterns from notes
for track in midi_file.tracks:
for msg in track:
if msg.type == 'note_on' and msg.velocity > 0:
note_counts[msg.time] += 1 # Number of notes by hour

# Inferring song structure based on patterns
print("Note occurrence frequency by hour:", note_counts)

By utilizing the code in Table 3, the structure of the song (Verse, Chorus, Bridge, etc.) can be analyzed and predicted through pattern analysis. Of course, various methods can be used in addition to the above-described methods. ④ The song analysis unit (310) can extract the pitch range by calculating the range of the lowest and highest notes of the melody part corresponding to the vocal part in the MIDI file. After extracting the pitch value of each note, the pitch range can be determined by finding the lowest and highest notes. Each note number in the MIDI file consists of 0 (C-1) to 123 (G9), and each note number has the same meaning as in Table 4 below.

Note number Note name octave 21 A0 the lowest piano note 60 C4 Middle C 127 G9 highest note

At this point, the pitch range can be extracted by loading note data from a MIDI file and then finding the lowest and highest notes. Alternatively, the pitch range can be automatically extracted using Python with code like that in Table 5.

import mido

# Importing MIDI files
midi_file = mido.MidiFile('example.mid')

# Reset note list
notes = []

# Extract notes from MIDI files
for track in midi_file.tracks:
for msg in track:
if msg.type == 'note_on' and msg.velocity > 0: # Only active notes
notes.append(msg.note)

# Calculating the range
if notes:
lowest_note = min(notes)
highest_note = max(notes)
print(f"Lowest note: {lowest_note}(MIDI Note), Highest note: {highest_note}(MIDI Note)")
else:
print("There are no notes in the MIDI file.")

⑤ The song analysis unit (310) can classify genres using a genre classification model that has been built based on indices including tempo, rhythm pattern, and chord progression in a MIDI file. For example, the genre classification model that has been built can be a machine learning model, for example, a support vector machine (SVM). Genres can be classified using tempo, rhythm pattern, chord progression, etc. as features. At this time, classification according to tempo, rhythm pattern, and chord progression can be as in Table 6, for example, but is not limited thereto.

Genre classification Tempo (BPM) Analysis -If the tempo is fast, the possibility of EDM, rock, and metal increases
-If the tempo is slow, the possibility of ballads and classical music increases Rhythm pattern analysis -If there are a lot of 4/4 beats, it's pop, rock, EDM
-If there are many 3/4, 6/8 beats, it is classical, jazz, waltz Code progression analysis -I-IV-VI (major progression) → pop, rock
-ii-VI (jazz progression) → jazz, blues
-i-VI-III-VII (minor progression) → Latin, EDM Analysis of instrument use -Music with strong drums → rock, metal, EDM
- Lots of pianos and string instruments → Classical, jazz Note density and distribution analysis - Lots of fast, short notes → trap, EDM, jazz
- Long sustained notes → Classical, ballad

<Similarity measurement>

The setting unit (320) measures the similarity between a vector embedding MIDI data of a pre-built MIDI data database and a vector embedding feature data, and then extracts similar MIDI data based on the similarity, and can set the pattern of values of instruments, equalizers, and filters applied to the similar MIDI data as candidate MIDI data. At this time, the MIDI data database can include, for a pre-built MIDI sound source, a MIDI file including MIDI data and lyric data, metadata for the MIDI file, settings for an actually applied instrument effect chain, and data on public response and public reaction to the MIDI sound source.

MIDI data database MIDI file MIDI data Lyrics data MetaData Instrument Effect Chain Settings Public Response and Public Reaction Data

Vector Database

A media database can be a vector database, which stores information as vectors. A vector, also known as a vector embedding, is a numerical representation of a data object. The powerful capabilities of vector embeddings can be leveraged to index and search large datasets composed of unstructured and semi-structured data, such as images, text, or sensor data. Because vector databases are built to manage vector embeddings, they offer a complete solution for managing unstructured and semi-structured data. Unlike vector search libraries or vector indexes, vector databases are data management solutions that enable metadata storage and filtering, are scalable, allow dynamic data modification, perform backups, and provide security features. Furthermore, vector databases can organize data through high-dimensional vectors, which can contain hundreds of dimensions, each corresponding to a specific feature or attribute of a data object.

Here, vector embeddings are numerical representations of topics, words, images, or other data. Vector embeddings, also known as embeddings, are generated by large-scale language models (LLMs) and other AI models. The distance between each vector embedding allows vector databases or vector search engines to determine similarity between vectors. Distances can represent multiple dimensions of data objects, allowing machine learning and AI to understand patterns, relationships, and underlying structures. Vector databases work by indexing and querying vector embeddings using algorithms. These algorithms enable approximate nearest neighbor (ANN) retrieval through hashing, quantization, or graph-based search. To retrieve information, ANN retrieval finds the closest vector equivalent of a query. However, because ANN retrieval is less computationally intensive than kNN retrieval (also known as nearest neighbor or k-nearest neighbor algorithm), approximate nearest neighbor retrieval is also less accurate. However, for large datasets of high-dimensional vectors, it operates efficiently and scalably, and the vector database pipeline can be as shown in Table 8 below. Further details are available in the public domain, so further explanation is omitted.

index Hashing algorithms, such as the Locality-Sensitive Hashing (LSH) algorithm, are best suited for approximate nearest neighbor retrieval because they provide fast results and produce approximate results. LSH uses a hash table, like a Sudoku puzzle, to map nearest neighbors. The query is hashed into a table and then compared to a set of vectors in the same table to determine similarity. Techniques like [quantization] (Product Quantization, PQ) break a vector into smaller parts, represent those parts as codes, and then reassemble those parts. The result is a vector and its component code representations. This ensemble of codes is called a codebook. When a query is made, a vector database using quantization breaks the query into codes and then matches them against the codebook to find the most similar code, producing a result. Graph algorithms , such as the Hierarchical Navigable Small World (HNSW) algorithm, use nodes to represent vectors. They cluster nodes and draw lines or edges between similar nodes to create a hierarchical graph. When a query is initiated, the algorithm traverses the graph hierarchy to find the node containing the vector most similar to the query vector. query Cosine Similarity sets the similarity between two vectors in a vector space, measuring the cosine of the angle between them. It determines whether vectors are opposite (represented as -1), orthogonal (represented as 0), or identical (represented as 1). [Euclidean Distance] measures the straight-line distance between vectors to determine similarity in a range from 0 to infinity. Identical vectors are represented as 0, and the larger the value, the greater the difference between vectors. The Dot Product similarity measure determines vector similarity over a range from minus infinity to infinity. The dot product measures the product of the magnitudes of two vectors and the cosine of the angle between them. It assigns a negative value to vectors that are far apart, a zero value to vectors that are orthogonal, and a positive value to vectors pointing in the same direction. Post-processing The final stage of a vector database pipeline is sometimes postprocessing or postfiltering, during which the vector database re-ranks the closest items using a different similarity measure. In this step, the database filters the closest items identified in the search query based on metadata.

At this time, the concepts of MIDI files, MIDI data, lyric data, MIDI sound sources, and metadata are organized and compared as shown in Table 9.

item explanation Inclusion form / example Whether sound is included Main uses MIDI file Digital file containing performance information (MIDI data + metadata, etc.) .mid, .kar files doesn't exist Accompaniment playback, composition, sheet music, karaoke, etc. MIDI data Actual note, instrument, and performance information contained within the MIDI file Note On/Off, Control Change, Program Change doesn't exist The core of music playback: performance instruction information Lyrics data Song lyrics text (syncable), used for subtitles Lyrics events in .txt, .lrc, and MIDI files doesn't exist Show subtitles, sing along MIDI sound source Playing MIDI data to create a synthetic sound that can actually be heard (accompaniment that can be heard by the human ear) .mp3, .wav, real-time output sound There is Provide accompaniment to users and generate sound sources Metadata Additional descriptive information within the MIDI file (tempo, track name, markers, lyrics, etc.) Meta Events: Set Tempo, Marker, Lyrics, etc. doesn't exist Song information description, visual distinction, analysis, etc.

At this time, a MIDI file may be a combination of MIDI data, a standard for outputting sound through an external device, and text, which is lyric data. The metadata of a MIDI file is data that includes basic information about the song (title, composer, tempo, key signature, etc.) and performance instructions (track name, lyrics, markers, etc.). Metadata is mainly included in metaevents, and the main items are as shown in Table 10 below.

Metadata items explanation Song Title (Track Name, Sequence Name) Song title or track name Composer (Copyright Notice) Composer, Copyright Information Tempo Song speed (BPM) information Key Signature The key of the song (e.g. C Major, A Minor) Time Signature The time signature of the song (e.g. 4/4, 3/4) Marker Annotations for specific points (e.g., "Verse Start") Lyrics Lyrics data in MIDI files Cue Point Points to synchronize with video or other events

Also, an instrument effect chain refers to a method of modifying a sound by connecting various audio effects applied to an instrument (especially an accompaniment sound) in a specific order. The components of an instrument effect chain can be, for example, as shown in Table 10.

Instrument effects chain explanation Equalizer (EQ) Change the tone by adjusting specific frequencies Reverb Create a sense of space by adding resonance effects Chorus Slightly overlap the sounds to create a richer feel Delay Repeats a sound at specific intervals to create an echo effect Distortion An effect that adds a harsh tone to instruments such as guitars. Compressor Maintain consistent sound by adjusting volume dynamics

Since the sound of the instrument effect chain changes depending on the connection order, different manufacturers of karaoke accompaniment machines can apply different settings.

In one embodiment of the present invention, the similarity between the song information of the MIDI file of the original music file and the song information of the previously constructed MIDI data database is measured. At this time, in one embodiment of the present invention, about 60,000 songs are constructed in the MIDI data database, and the song information of this MIDI data database is extracted and compared with the original music file, thereby extracting N MIDI data of songs having song information similar to the original music file. This is called similar MIDI data. In the N similar MIDI data listed in this way in order of extraction and similarity, the patterns of the values of instruments, equalizers, and filters can be set as candidate MIDI data. In addition, the instrument, equalizer (EQ), and filter settings used in the similar MIDI data can be set as candidate MIDI data. In other words, after extracting a pattern from a sound source similar to the original music file, this pattern is selected as the MIDI data of the original music file, and a process of refining it is performed.

Sound Optimization Process

The generation unit (330) can generate MIDI data for the original music file by performing a preset sound optimization process on candidate MIDI data. An additional rule of prioritizing vocal clarity can be applied to the candidate MIDI data extracted using the above-described method, drums and bass can be processed relatively clearly in consideration of the characteristics of karaoke, and logic can be applied (duckling technique) to automatically attenuate the volume of high-frequency instruments when the chorus section or vocals are in the high-frequency range to minimize vocal interference. In addition, an effect chain can be automatically assigned to optimize genre-specific presets to capture genre-specific timbres or tones.

Here, the additional rule of vocal clarity priority refers to an audio processing method that adjusts the vocals (the singer's voice) to be more clearly audible when accompaniment and vocals are output together on a karaoke accompaniment device (400). The following principles and techniques are applied to achieve this.

Key Rules for Improving Vocal Clarity Equalizer (EQ) adjustment - Emphasize vocal frequencies → Boost the 1kHz to 5kHz band to enhance the clarity of your voice.
- Accompaniment low-frequency attenuation → Slightly reduces low frequencies below 200 Hz to prevent vocals from being buried.
- Accompaniment high-frequency attenuation → Slightly reduce above 5kHz to prevent clashing between instruments and vocals Dynamic processing -Compressor → Maintains a constant vocal volume so it doesn't get buried
-Ducking → A function that automatically lowers the accompaniment volume when vocals come in. Adjust stereo balance -The accompaniment is spread out to the left and right, and the vocals are positioned in the center to make them more clearly audible. Minimize reverb and delay
- Reduce the spatial effects (reverb, delay) of the accompaniment to separate it from the vocals.
-Apply appropriate reverb to vocals to maintain natural resonance. Automatic accompaniment volume adjustment
- Karaoke machines sometimes use vocal detection sensors to adjust the accompaniment based on the volume of the vocals.

Also, in order to suit the karaoke environment, the low sounds must be emphasized, so low-frequency instruments such as drums and bass can be processed clearly, and when the vocal melody is high-frequency, the high-frequency instruments can be reduced using the ducking technique. As mentioned above, the ducking technique is a technology that automatically lowers the volume of other sounds when a specific sound is input in audio mixing. The principle of ducking is that when a trigger sound is detected, the volume of the target sound is automatically lowered, and when the trigger sound disappears, the volume of the target sound is restored to the original. This technique is mainly used to balance the vocals and accompaniment. Accordingly, it is often used to automatically adjust the accompaniment so that it does not cover the vocals, and it can be said to be a technique that automatically adjusts the volume to emphasize important sounds.

In summary, the preset sound optimization process may include a step of applying a preset vocal clarity priority rule to candidate MIDI data, a step of setting the candidate MIDI data to make drums and bass clearly audible, a step of automatically attenuating the volume of at least one preset instrument sound in a chorus section or a high-frequency section of the vocal by using a ducking technique on the candidate MIDI data, and a step of automatically assigning an effect chain of the candidate MIDI data to correspond to the genre of the original music file. Accordingly, by including each data extracted in <song information analysis>, <candidate MIDI data extraction>, and <sound optimization process> as MIDI data, MIDI data can be generated.

However, since the MIDI data generated in this way is generated based on existing MIDI data, that is, candidate MIDI data similar to karaoke accompaniment (MR), MIDI data that exactly matches the original music file may not be generated, and inappropriate equalizer settings, instrument selection, or filter settings may be included. Accordingly, the user terminal (100) may ultimately inspect this, and MIDI data that passes the inspection may be provided as the final MIDI data.

<Play Accompaniment>

The accompaniment playback unit (340) can input a MIDI file in which lyric data corresponding to the MIDI data is combined with the MIDI data into the karaoke accompaniment machine (400) and play it when MIDI data is generated in the generation unit (330). The accompaniment playback unit (340) can extract instrument data from the MIDI data of the MIDI file, extract instrument sounds from the instrument data, and extract MIDI messages from the MIDI data to play the MIDI file in the karaoke accompaniment machine (400). The MIDI data of the present invention is set to be output not only in the karaoke accompaniment machine (400) but also in various devices. For detailed information about this, please refer to the Korean Patent Registration No. 10-1881854 (published on July 25, 2018), which is a registered patent of the present applicant.

Karaoke Song Recommendations

In addition, based on the MIDI data database according to one embodiment of the present invention, a song or an appropriate key that fits the vocal range of a singer can be recommended. At this time, if the vocal range or key is focused, the preference of the singer can be excluded, so a recommendation system that can consider the preference, such as collaborative filtering, can be further added. At this time, in order to identify the vocal range of the singer, the user's vocal range is quantitatively compared with the probability distribution of the MIDI sound source of the karaoke accompaniment device (400) using the Kullback-Leibler Divergence, and a recommendation can be made considering the user's preference among the most similar MIDI sound sources. The MIDI sound source is the actual sound heard when a MIDI file is played.

Kullback-Leibler divergence, one of the important concepts in information theory, is used to measure the similarity between two probability distributions. Kullback-Leibler divergence is used to measure the difference between two probability distributions P and Q to be compared, and outputs a value closer to 0 as the similarity between the two distributions increases. In mathematical equation 1, which calculates the difference between two probability distributions, x is an index for each value of the random variable, and is summed over all possible values of the distribution. P(x) and Q(x) are the probabilities of the random variable x, respectively. In one embodiment of the present invention, Kullback-Leibler divergence is used from the perspective of evaluating whether a user is suitable for singing a song and providing feedback.

This characteristic of Kullback-Leibler divergence quantitatively indicates the basis for songs suitable for the user's vocal range, and is suitable for recommending songs or suggesting criteria for key settings. In mathematical expression 1, P represents the probability distribution of the vocal range data input by the user, and Q represents the probability distribution of the karaoke sheet music data. If the key adjustment range is within 6 degrees up and down that is possible with a karaoke accompaniment machine (400) according to the Kullback-Leibler divergence result, the song can be applied as a recommended song. Among the songs in the recommended song range, songs with a smaller Kullback-Leibler divergence value are judged to have a higher suitability with the user, and thus can be recommended preferentially.

Song Generation Using Generative AI

Recently, generative AI has been generating not only text but also songs, but there are phenomena such as the progression of the song being harmonically unstable or the sound jumping around. To solve this, based on a MIDI data database (vector database) according to an embodiment of the present invention, Bi-LSTM (Bidirectional Long Short-Term Memory) is used to learn the development, song structure, and range of the song, and a song can be generated based on this. At this time, Bi-LSTM is an extended version of LSTM (Long Short-Term Memory) and is a neural network model that learns more context information using two LSTMs, one forward and one backward. At this time, in the MIDI file, the high note clef and the low note clef are separately learned on two channels.

The existing single-layer-based learning centered on piano rolls only extracts melody scales from MIDI files and learns them, which limits the richness of the generated songs. However, Bi-LSTM inputs data in sequence order during forward learning, and in the reverse order of the sequence during backward learning, so the output is a single output that concatenates the forward and reverse outputs. An attention layer can be applied for this connection, which reflects the weights of influential elements among the encoder's input data. The learned data is concatenated and collected in the merge layer. Finally, the normalization process is performed to produce the final output.

By extracting notes, note durations, rests, rest durations, and chords from MIDI files by distinguishing between high and low notes, and applying them to an attention-based Bi-LSTM for training, we can generate notes and chords that fit the development of the song, and create harmonically stable songs. Of course, it is self-evident that the above-described MIDI data database can be trained in various ways other than by training it with the attention-based Bi-LSTM described above.

Hereinafter, the operation process according to the configuration of the extraction service providing server of FIG. 2 described above will be described in detail with reference to FIGS. 3 and 4 as examples. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to FIG. 3, (a) the extraction service providing server (300) can convert an original music file into a MIDI file using AI, and perform song analysis using the MIDI file as in (b). Through this, after extracting tempo, key/key, song structure, range, and genre, (c) similar MIDI data having similar tempo, key/key, song structure, range, and genre can be extracted from a pre-built MIDI data database, and (d) candidate MIDI data can be extracted by identifying the pattern of the instrument, equalizer, and filter values of the similar MIDI data. Then, the extraction service providing server (300) can generate an accompaniment that is easy to sing in a karaoke room through a sound optimization process as in (a) of FIG. 4. The extraction service providing server (300) can (b) gather all of these respective data (song structure analysis-similarity identification-sound optimization) to generate MIDI data.

The extraction service providing server (300), (c) if MIDI data is generated in this way, adds lyric data to create a MIDI file, and extracts instrument data of the MIDI file using this. At this time, it can be extracted from the sound library of the synth module of the karaoke accompaniment machine (400) and loaded into memory, and a MIDI message can be extracted from the MIDI data and played. For details on this, refer to the aforementioned Korean registered patent of the present applicant. The MIDI file played in this way can be finalized or edited and refined in (d) the user terminal (100).

Matters not described in the AI-based midi data extraction service providing method of FIGS. 2 to 4 are the same as or can be easily inferred from the contents described in the AI-based midi data extraction service providing method of FIG. 1 above, and therefore, description thereof will be omitted below.

FIG. 5 is a diagram illustrating a process of transmitting and receiving data between each component included in the AI-based media data extraction solution providing system of FIG. 1 according to one embodiment of the present invention. Hereinafter, an example of a process of transmitting and receiving data between each component will be described through FIG. 5, but the present invention is not limited to this embodiment, and it will be apparent to those skilled in the art that the process of transmitting and receiving data illustrated in FIG. 5 may be modified according to various embodiments described above.

Referring to FIG. 5, the extraction service providing server inputs an original music file into a pre-built AI, converts it into a MIDI file, and then extracts tempo, key, composition, song structure, range, and genre as feature data from the MIDI file (S5100).

Then, the extraction service providing server measures the similarity between a vector embedding MIDI data of a pre-built MIDI data database and a vector embedding feature data, extracts similar MIDI data based on the similarity, and sets the pattern of values of instruments, equalizers, and filters applied to the similar MIDI data as candidate MIDI data (S5200).

In addition, the extraction service providing server generates MIDI data for the original music file by performing a preset sound optimization process on the candidate MIDI data (S5300).

The order of the above-described steps (S5100 to S5300) is merely an example and is not limited thereto. That is, the order of the above-described steps (S5100 to S5300) may be mutually changed, and some of the steps may be executed simultaneously or deleted.

Matters not described in the AI-based midi data extraction service providing method of FIG. 5 are the same as or can be easily inferred from the contents described in the AI-based midi data extraction service providing method through FIGS. 1 to 4 above, and therefore, description thereof will be omitted below.

The method for providing an AI-based media data extraction service according to one embodiment described through FIG. 5 may also be implemented in the form of a recording medium including computer-executable instructions, such as an application or program module executed by a computer. The computer-readable medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include all computer storage media. The computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

The method for providing an AI-based midi data extraction service according to an embodiment of the present invention described above can be executed by an application that is installed by default on a terminal (which may include a program included in a platform or operating system that is installed by default on the terminal), and can also be executed by an application (i.e., a program) that a user directly installs on a master terminal through an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the method for providing an AI-based midi data extraction service according to an embodiment of the present invention described above can be implemented as an application (i.e., a program) that is installed by default on a terminal or directly installed by a user, and can be recorded on a computer-readable recording medium such as a terminal.

The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims (6)

A user terminal for selecting an original music file to generate MIDI data of a MIDI file to be input into a karaoke accompaniment machine; and
An extraction service providing server including a song analysis unit that converts the original music file into a MIDI file by inputting it into a pre-built AI and then extracts tempo, key, composition, song structure, range, and genre as feature data from the MIDI file, a setting unit that measures the similarity between a vector embedding MIDI data of a pre-built MIDI data database and a vector embedding the feature data, and then extracts similar MIDI data based on the similarity, and sets the pattern of values of instruments, equalizers, and filters applied to the similar MIDI data as candidate MIDI data, and a generation unit that generates MIDI data for the original music file by performing a pre-built sound optimization process on the candidate MIDI data;
A system that provides an AI-based media data extraction solution including .
In the first paragraph,
The above song analysis section,
When analyzing the above tempo, the tempo meta event within the MIDI file is extracted to analyze the tempo.
By identifying the most frequent notes and chord progression patterns through the note distribution of the above MIDI file, the key and composition are analyzed.
In the above MIDI file, the sections corresponding to the intro, verse, chorus, interlude, and ending are analyzed using the established song structure analysis model.
Extract the range of the lowest and highest notes of the melody part corresponding to the vocal part in the above MIDI file, and
A system providing an AI-based MIDI data extraction solution characterized in that it classifies genres using a genre classification model established based on indices including tempo, rhythm pattern, and chord progression in the above MIDI file.
In the first paragraph,
The above MIDI data database is,
Regarding MIDI sound source, which is the actual sound heard by playing a MIDI file, a MIDI file including MIDI data and lyrics data;
Metadata for the above MIDI file;
Setting the instrument effect chain actually applied to the above MIDI file;
Public response and reaction data for the above MIDI sound source;
A system providing an AI-based media data extraction solution, characterized by including:
In the first paragraph,
The above preset sound optimization process is:
A step of applying a preset vocal clarity priority rule to the above candidate MIDI data;
A step of setting the drums and bass in the above candidate MIDI data to be clearly audible;
A step of automatically attenuating the volume of at least one instrument sound preset in the chorus section or high-frequency section of the vocal using the duckling technique in the above candidate MIDI data;
A step of automatically assigning an effect chain of the candidate MIDI data to correspond to the genre of the original music file;
A system providing an AI-based media data extraction solution, characterized by including:
In the first paragraph,
The above extraction service providing server is,
When the MIDI data is generated in the above generation unit, an accompaniment playback unit that inputs a MIDI file in which the lyrics data corresponding to the MIDI data is combined with the MIDI data into the karaoke accompaniment unit and plays it;
A system providing an AI-based media data extraction solution characterized by further including:
In paragraph 5,
The above accompaniment playback section is,
A system for providing an AI-based MIDI data extraction solution, characterized in that it extracts instrument data from the MIDI data of the MIDI file, extracts instrument sounds from the instrument data, and causes the MIDI file to be played in the karaoke accompaniment machine.