KR102721179B1 - Method and system for standardizing text data on food material - Google Patents

Abstract

According to one aspect of the present invention, a method for standardizing text data regarding a foodstuff is provided, comprising: a step of tokenizing text data regarding a foodstuff into a plurality of tokens; a step of determining a weight of each of the plurality of tokens using an artificial intelligence-based token weight determination model, and calculating a food ingredient name similarity, which is a degree of similarity between each of a plurality of standard food ingredient names included in standard food ingredient data and a food ingredient name included in the text data, based on the weight; and a step of standardizing the text data regarding a foodstuff by determining a target food ingredient name corresponding to the text data from among the plurality of standard food ingredient names based on the food ingredient name similarity.

Description

METHOD AND SYSTEM FOR STANDARDIZING TEXT DATA ON FOOD MATERIAL

The present invention relates to a method and system for standardizing text data regarding food materials.

By statistically analyzing the distribution information of food materials (e.g., food material purchase history information) of various stores that handle food materials, for example, stores that make and sell food using food materials or provide services related to food sales, various useful information can be obtained, such as information on the demand and supply of various food materials, information on market conditions such as prices, consumer sentiment, and trends. In addition, in order to increase the reliability of such statistical analysis, it is essential to first accurately collect data on food materials.

However, since there is no policy-based labeling method for food ingredients, the same food ingredients are often distributed labeled in various ways. In fact, when purchasing a food ingredient called A, it is very common for one buyer to order 50 items with the name A', while another buyer may order 50 items with the name A''. In such cases, it is difficult to conduct accurate statistical analysis because it may be counted that 50 items of food ingredient A' and 50 items of food ingredient A'' were ordered, rather than 100 items of food ingredient A being ordered. Of course, this problem can be solved by investing manpower, but it is not a good solution in reality because it requires continuous investment of time and money.

Accordingly, the inventor(s) of the present invention propose a technology that enables accurate statistical analysis of the distribution of food materials by standardizing text data on food materials.

Korean Patent Publication No. 10-1975911 (April 30, 2019)

The purpose of the present invention is to solve all of the problems of the above-mentioned prior art.

In addition, the present invention has another purpose of standardizing text data about foodstuffs by tokenizing text data about foodstuffs into a plurality of tokens, determining a weight for each of the plurality of tokens using an artificial intelligence-based token weight determination model, calculating a food ingredient name similarity, which is a degree of similarity between each of a plurality of standard food ingredient names included in standard foodstuff data and a food ingredient name included in text data, based on the weights, and determining a target food ingredient name corresponding to the text data from among the plurality of standard food ingredient names based on the food ingredient name similarity.

In addition, another purpose of the present invention is to enable accurate statistical analysis of the distribution of food materials.

A representative configuration of the present invention to achieve the above purpose is as follows.

According to one aspect of the present invention, a method is provided, including: a step of tokenizing text data regarding a food ingredient into a plurality of tokens; a step of determining a weight of each of the plurality of tokens using an artificial intelligence-based token weight determination model; and a step of calculating a food ingredient name similarity, which is a degree of similarity between each of a plurality of standard food ingredient names included in standard food ingredient data and a food ingredient name included in the text data, based on the weights; and a step of standardizing the text data regarding the food ingredient by determining a target food ingredient name corresponding to the text data from among the plurality of standard food ingredient names based on the food ingredient name similarity.

According to another aspect of the present invention, a data tokenization unit for tokenizing text data regarding food materials into a plurality of tokens;

A system is provided, comprising: a similarity calculation unit that determines the weight of each of the plurality of tokens using an artificial intelligence-based token weight determination model, and calculates a food ingredient name similarity, which is a degree of similarity between each of the plurality of standard food ingredient names included in standard food ingredient data and the food ingredient name included in the text data, based on the weights; and a data standardization unit that standardizes text data regarding the food ingredient by determining a target food ingredient name corresponding to the text data from among the plurality of standard food ingredient names based on the food ingredient name similarity.

In addition, other methods for implementing the present invention, other systems, and non-transitory computer-readable recording media recording a computer program for executing the above methods are further provided.

According to the present invention, text data regarding foodstuffs is tokenized into a plurality of tokens, a weight of each of the plurality of tokens is determined using an artificial intelligence-based token weight determination model, and based on the weights, a food ingredient name similarity, which is a similarity between each of a plurality of standard food ingredient names included in standard food ingredient data and a food ingredient name included in text data, is calculated, and based on the food ingredient name similarity, a target food ingredient name corresponding to the text data is determined from among the plurality of standard food ingredient names, thereby making it possible to standardize text data regarding foodstuffs.

In addition, according to the present invention, statistical analysis regarding the distribution of food materials can be performed accurately.

FIG. 1 is a diagram schematically illustrating the configuration of an entire system for standardizing text data regarding foodstuffs according to one embodiment of the present invention.
FIG. 2 is a drawing detailing the internal configuration of a standardization system according to one embodiment of the present invention.
FIG. 3 is a diagram exemplarily showing a process of standardizing text data regarding food materials according to one embodiment of the present invention.

The detailed description of the present invention set forth below refers to the accompanying drawings which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention, while different from each other, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be modified and implemented from one embodiment to another without departing from the spirit and scope of the invention. It should also be understood that the positions or arrangements of individual components within each embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention is to be taken to encompass the scope of the claims and all equivalents thereof. Like reference numerals in the drawings represent the same or similar elements throughout the several aspects.

Hereinafter, various preferred embodiments of the present invention will be described in detail with reference to the attached drawings so that a person having ordinary skill in the art to which the present invention pertains can easily practice the present invention.

In this specification, the term “food material” should be understood as a broad concept that includes not only the materials used to make food, but also the food made using those materials.

Composition of the entire system

FIG. 1 is a diagram schematically illustrating the configuration of an entire system for standardizing text data regarding foodstuffs according to one embodiment of the present invention.

As illustrated in FIG. 1, the entire system according to one embodiment of the present invention may include a communication network (100), a standardization system (200), and a device (300).

First, a communication network (100) according to an embodiment of the present invention may be configured regardless of a communication mode such as wired communication or wireless communication, and may be configured with various communication networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). Preferably, the communication network (100) referred to in the present specification may be the well-known Internet or the World Wide Web (WWW). However, the communication network (100) is not necessarily limited thereto, and may include at least a part of a well-known wired and wireless data communication network, a well-known telephone network, or a well-known wired and wireless television communication network.

For example, the communication network (100) may be a wireless data communication network that implements, at least in part, a conventional communication method such as WiFi communication, WiFi-Direct communication, Long Term Evolution (LTE) communication, 5G communication, Bluetooth communication (including Bluetooth Low Energy (BLE) communication), infrared communication, or ultrasonic communication. As another example, the communication network (100) may be an optical communication network that implements, at least in part, a conventional communication method such as LiFi (Light Fidelity).

Next, a standardization system (200) according to one embodiment of the present invention tokenizes text data regarding foodstuffs into a plurality of tokens, determines a weight for each of the plurality of tokens using an artificial intelligence-based token weight determination model, calculates a food ingredient name similarity, which is a similarity between each of a plurality of standard food ingredient names included in standard food ingredient data and a food ingredient name included in text data, based on the weights, and determines a target food ingredient name corresponding to the text data from among the plurality of standard food ingredient names based on the food ingredient name similarity, thereby performing a function of standardizing text data regarding foodstuffs.

The configuration and function of the standardization system (200) according to the present invention will be described in detail below.

Next, a device (300) according to one embodiment of the present invention is a digital device that includes a function for communicating after being connected to a standardization system (200), and any digital device equipped with a memory means and a microprocessor and having a computing capability, such as a smart phone, a tablet, a smart watch, a smart band, smart glasses, a desktop computer, a notebook computer, a workstation, a PDA, a web pad, a mobile phone, etc., can be adopted as the device (300) according to the present invention.

In particular, the device (300) may include an application (not shown) that supports a user to receive a service according to the present invention from the standardization system (200). Such an application may be downloaded from the standardization system (200) or an external application distribution server (not shown). Meanwhile, the nature of such an application may be generally similar to the data tokenization unit (210), the similarity calculation unit (220), the data standardization unit (230), the communication unit (240), and the control unit (250) of the standardization system (200) as described below. Here, at least a part of the application may be replaced with a hardware device or firmware device that can perform functions substantially identical to or equivalent thereto, as necessary.

Composition of the standardization system

Below, the internal configuration of the standardization system (200) that performs important functions for implementing the present invention and the functions of each component will be examined.

FIG. 2 is a drawing detailing the internal configuration of a standardization system (200) according to one embodiment of the present invention.

As illustrated in FIG. 2, a standardization system (200) according to one embodiment of the present invention may be configured to include a data tokenization unit (210), a similarity calculation unit (220), a data standardization unit (230), a communication unit (240), and a control unit (250). According to one embodiment of the present invention, at least some of the data tokenization unit (210), the similarity calculation unit (220), the data standardization unit (230), the communication unit (240), and the control unit (250) may be program modules that communicate with an external system (not shown). Such program modules may be included in the standardization system (200) in the form of an operating system, an application program module, or other program modules, and may be physically stored in various known memory devices. In addition, such program modules may be stored in a remote memory device that can communicate with the standardization system (200). Meanwhile, these program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform specific tasks or execute specific abstract data types according to the present invention.

Meanwhile, although the standardization system (200) has been described as above, this description is exemplary, and it is obvious to those skilled in the art that at least some of the components or functions of the standardization system (200) may be realized within a device (300) or a server (not shown) or included within an external system (not shown) as needed.

First, the data tokenization unit (210) according to one embodiment of the present invention can perform a function of tokenizing text data regarding food materials into a plurality of tokens.

Specifically, the data tokenization unit (210) according to one embodiment of the present invention can obtain text data regarding food ingredients for tokenization. According to one embodiment of the present invention, text data regarding food ingredients can mean various text data that can be collected during the process of producing, processing, distributing, etc., the food ingredients.

For example, according to one embodiment of the present invention, data such as the name of the food ingredient, unit information, etc., which a purchaser of the food ingredient inputs or selects to order the food ingredient during the distribution process of the food ingredient, may be included in such text data. Here, according to one embodiment of the present invention, the name of the food ingredient may include information such as the category of the food ingredient, brand, distributor, etc., and the unit information may include information regarding the amount of the corresponding food ingredient (e.g., weight, volume, (unit) quantity, packaging unit, etc.).

Meanwhile, according to one embodiment of the present invention, text data regarding food ingredients may be acquired in a text format from the beginning, but may also be acquired by converting data of another format acquired using a technology such as STT (Speech-To-Text) into a text format. However, the type of text data regarding food ingredients according to one embodiment of the present invention or the method of acquiring the same are not limited to the contents described above, and may be variously changed within the scope that can achieve the purpose of the present invention.

FIG. 3 is a drawing exemplarily showing a process of standardizing text data regarding food materials according to one embodiment of the present invention.

Referring to FIG. 3, an example of text data about a food ingredient that can be obtained according to one embodiment of the present invention, such as "Sase Chicken Tender 1 kg" (410), is illustrated in a column named input_name. As illustrated, the text data about the food ingredient includes a food ingredient name (e.g., "Sase Chicken Tender", "Hanil Original Dried Seaweed Soup", "Sase Tuna Can", "Sase Salkoki Tuna", "Fanta (Orange)", and "Fanta (Pine)"). And, depending on the case, the text data about the food ingredient may further include information about a unit, that is, unit information (e.g., "1 kg", "1.88 kg", "1.88 kg 6 pieces", and "355 ml x 24 cans").

Continuing, the data tokenization unit (210) according to one embodiment of the present invention can tokenize the text data about the food material obtained as described above into a plurality of tokens. According to one embodiment of the present invention, tokenization may mean a process of dividing the text data about the food material into a plurality of pieces in order to calculate the food material similarity and/or the unit similarity described below, and each of the divided pieces may correspond to a token. The data tokenization unit (210) according to one embodiment of the present invention may perform such tokenization based on a pattern of the text data about the food material, for example, a pattern such as that tokens about unit information are mainly located at the end of the text data or are located around a comma or a multiplication mark. In addition, the data tokenization unit (210) according to one embodiment of the present invention can process the text data about the food material using various types of algorithms related to natural language processing, text mining, etc.

For example, the data tokenization unit (210) according to one embodiment of the present invention can tokenize text data regarding food ingredients into two or more tokens, and the two or more tokens can correspond to at least one type among food ingredient category, brand, distributor, weight, volume, quantity, and packaging unit. At this time, it should be understood that tokenization may be performed in a state where two or more tokens of the same type are present. For example, there may be two or more food ingredient category tokens or two or more brand tokens. In this case where there are two or more tokens of the same type, the tokens of the same type may correspond to upper and lower concepts or may correspond to parallel concepts.

For example, referring to FIG. 3, the text data regarding food material, “Sase Chicken Tender 1kg” (410), may be tokenized into three tokens: “Sase”, “Chicken Tender”, and “1kg”.

However, tokenization according to one embodiment of the present invention is not limited to the contents described above, and may be variously changed within the scope that can achieve the purpose of the present invention.

Next, the similarity calculation unit (220) according to one embodiment of the present invention can perform a function of determining the weight of each of a plurality of tokens using an artificial intelligence-based token weight determination model.

Specifically, when text data regarding food ingredients are tokenized into a plurality of tokens by the data tokenization unit (210) according to one embodiment of the present invention, the similarity calculation unit (220) according to one embodiment of the present invention can use an artificial intelligence-based token weight determination model to assign differential weights to each token according to the importance of each of the plurality of tokens. Such weights can be used in calculating the similarity of food ingredient names, and a detailed description thereof will be provided later.

More specifically, according to one embodiment of the present invention, the token weight determination model can be trained to assign a higher weight to a token the more frequently it appears in a correct answer data set (i.e., a collection of data on target food ingredient names to which food ingredient names included in text data on food ingredients should correspond; this will be described in detail later) used for training the model. For example, the token weight determination model according to one embodiment of the present invention can be trained based on an information gain score so as to determine the weight in the above manner.

However, if the weight of each token is determined solely based on the frequency of its appearance in the correct data set, the results may not sufficiently take into account the specificity of the food domain. For example, if the token "moist" appears frequently in the correct data set and the weight is determined solely based on the frequency of its appearance, the token will be given a high weight, but in reality, since the token is a word that is included in too many food ingredient names without any special meaning, it may not be of much help in identifying the food ingredient, or rather, it may hinder the identification. Of course, the importance of a token is not something that can be uniformly determined, and even a token with low importance in general may be treated as a token with high importance when considering its relationship with other specific tokens (words).

Considering these circumstances, a token weight determination model according to one embodiment of the present invention may be learned based on ontology data regarding food ingredients. According to one embodiment of the present invention, ontology data regarding food ingredients may be data defining relationships between words regarding food ingredient categories, brands, distributors, weights, volumes, quantities, packaging units, etc., which may be included in text data regarding food ingredients, the importance of words according to the relationships, the importance according to the meaning of each word itself in the food ingredient domain, etc.

For example, in ontology data according to one embodiment of the present invention, a specific word may be defined to be treated as having high importance considering that the word is not interpreted according to its dictionary meaning in the foodstuff domain, but is a word that signifies a specific brand. In another example, in ontology data according to one embodiment of the present invention, a specific word may be defined to be treated as having high importance only when it is used together with another specific word.

According to one embodiment of the present invention, by allowing a token weight determination model to be learned based on ontology data regarding foodstuffs, more accurate weights can be assigned to tokens according to the importance of each token in the foodstuff domain.

Additionally, according to one embodiment of the present invention, the learning data of the token weight determination model may be augmented by hard negative sampling based on ontology data regarding food materials.

Specifically, according to one embodiment of the present invention, after a token weight determination model performs learning based on ontology data regarding foodstuffs as described above, when determining a target foodstuff name using the learned token weight determination model, the learning data of the model can be augmented using data that leads to an incorrect decision or data that is highly likely to lead to an incorrect decision. Then, by re-training the token weight determination model using the augmented learning data, the performance of the model can be improved.

A similarity calculation unit (220) according to one embodiment of the present invention can perform a function of calculating food ingredient name similarity, which is the similarity between each of a plurality of standard food ingredient names included in standard food ingredient data and food ingredient names included in text data about food ingredients, based on the weight of each token calculated as described above.

Specifically, according to one embodiment of the present invention, standard food ingredient data may mean a set of data to which text data regarding food ingredients should correspond (or be mapped). Referring to FIG. 3, standard food ingredient data may include a standard food ingredient name (420) and unit information (430) corresponding thereto. For example, "Sase Chicken Tender 1kg" (410), which is text data regarding food ingredients, may correspond to "Tender Stick (Sase, Chicken) 1kg", "ea", and "1000g" among the standard food ingredient data. As illustrated in FIG. 3, it should be understood that the standard food ingredient name (420) may include information regarding not only the food ingredient category, brand, and distributor, but also, in some cases, information regarding the weight, volume, quantity, and packaging unit of the food ingredient.

Continuing, the similarity calculation unit (220) according to one embodiment of the present invention can calculate the food ingredient name similarity based on the number or ratio of overlapping tokens between each of a plurality of standard food ingredient names (420) included in standard food ingredient data and the food ingredient name (410) included in text data regarding the food ingredient.

According to one embodiment of the present invention, the basic principle is that the more the number or the higher the ratio of overlapping tokens, the higher the similarity of the food ingredient name is calculated, but the importance of the overlapping tokens, i.e., the weight of the overlapping tokens, may be considered when calculating the similarity. Accordingly, even if the number of tokens overlapping between the food ingredient name (410) and the first standard food ingredient name is less than the number of tokens overlapping between the food ingredient name (410) and the second standard food ingredient name (or lower than the ratio), if the weight of the token(s) overlapping between the food ingredient name (410) and the first standard food ingredient name is high, the similarity of the food ingredient name with the first standard food ingredient name may be calculated to be higher than the similarity of the food ingredient name with the second standard food ingredient name.

A similarity calculation unit (220) according to one embodiment of the present invention can calculate the similarity of food ingredient names such that the influence of a noisy token among a plurality of tokens on the similarity of food ingredient names is smaller than the influence of a non-noisy token on the similarity of food ingredient names.

Specifically, according to one embodiment of the present invention, a plurality of tokens generated by the data tokenization unit (210) according to one embodiment of the present invention may include noisy tokens in some cases. According to one embodiment of the present invention, a noisy token may correspond to a token that renders the calculation result inaccurate when calculating the similarity of food ingredient names. Such noisy tokens may have a small influence on the similarity of food ingredients names by being given a low weight in the token weight determination process by the similarity calculation unit (220) according to one embodiment of the present invention. However, the influence on the similarity of food ingredients names may also be small by performing filtering separately from the weight of the tokens.

Meanwhile, the similarity calculation unit (220) according to one embodiment of the present invention can calculate unit similarity, which is the similarity between unit information of a food ingredient included in text data regarding the food ingredient and unit information corresponding to each of a plurality of standard food ingredient names included in standard food ingredient data.

Specifically described with reference to FIG. 3, the similarity calculation unit (220) according to one embodiment of the present invention can calculate the unit similarity between the unit information (“1 kg”) of the food ingredient included in the text data (410) regarding the food ingredient and the unit information (430) corresponding to each of the plurality of standard food ingredient names (420) included in the standard food ingredient data. At this time, it should be understood that according to one embodiment of the present invention, only a part of the unit information (430) may be used when calculating the unit similarity. For example, in the case of “tender stick (chicken, sachet) 1 kg”, among the unit information “ea” and “1000g” corresponding thereto, only “1000g” may be used when calculating the unit similarity with the unit information (“1 kg”) of the food ingredient included in the text data (410) regarding the food ingredient.

More specifically, the similarity calculation unit (220) according to one embodiment of the present invention can calculate the unit similarity as above based on the edit distance between the unit information of the food ingredient included in the text data regarding the food ingredient and the unit information corresponding to each of the plurality of standard food ingredient names included in the standard food ingredient data.

For example, a similarity calculation unit (220) according to one embodiment of the present invention can calculate a Damerau-Levenshtein distance (i.e., an edit distance) between unit information of a food ingredient included in text data regarding a food ingredient and unit information corresponding to each of a plurality of standard food ingredient names included in standard food ingredient data, and calculate a unit similarity such that the shorter the distance (the smaller the value), the higher the value.

In addition, the similarity calculation unit (220) according to one embodiment of the present invention can calculate the unit similarity so that the influence of the noisy token among the plurality of tokens on the unit similarity is smaller than the influence of the non-noisy token on the unit similarity. The method can be similar to processing the noisy and non-noisy tokens in the process of calculating the similarity of food ingredient names, and since the content regarding this has been described above, a duplicate description will be omitted.

Next, the data standardization unit (230) according to one embodiment of the present invention can perform a function of standardizing text data about a food ingredient by determining a target food ingredient name corresponding to text data about the food ingredient among a plurality of standard food ingredient names based on the similarity of the food ingredient names.

Specifically, the data standardization unit (230) according to one embodiment of the present invention can determine, as the target food ingredient name, a standard food ingredient name having the highest similarity with a food ingredient name included in text data about the food ingredient among a plurality of standard food ingredient names included in the standard food ingredient data.

In addition, the data standardization unit (230) according to one embodiment of the present invention can determine the target food ingredient name based not only on the food ingredient name similarity as described above, but also on the unit similarity, which is the similarity between the unit information of the food ingredient included in the text data regarding the food ingredient and the unit information corresponding to each of the plurality of standard food ingredient names included in the standard food ingredient data.

Specifically, according to one embodiment of the present invention, predetermined weights may be given to the food ingredient name similarity and unit similarity, and among a plurality of standard food ingredient names included in standard food ingredient data, the standard food ingredient name with the highest result of combining the food ingredient name similarity and unit similarity by considering the weights may be determined as the target food ingredient name. Meanwhile, it should be understood that the above predetermined weights are not uniformly determined, and may be determined differently for each product category.

By tokenizing text data about foodstuffs in this way, the similarity of food ingredient names and unit similarities are calculated separately, and the target food ingredient name is determined based on the result of combining these, more accurate results can be obtained than when text data about foodstuffs is utilized as a whole to calculate the similarity with standard food ingredient data.

Next, the communication unit (240) according to one embodiment of the present invention can perform a function that enables data transmission and reception from/to the data tokenization unit (210), the similarity calculation unit (220), and the data standardization unit (230).

Finally, the control unit (250) according to one embodiment of the present invention can perform a function of controlling the flow of data between the data tokenization unit (210), the similarity calculation unit (220), the data standardization unit (230), and the communication unit (240). That is, the control unit (250) according to one embodiment of the present invention can control the data tokenization unit (210), the similarity calculation unit (220), the data standardization unit (230), and the communication unit (240) to perform their own functions by controlling the flow of data from/to the outside of the standardization system (200) or the flow of data between each component of the standardization system (200).

The embodiments of the present invention described above may be implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the computer-readable recording medium may be those specially designed and configured for the present invention or those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices may be changed into one or more software modules to perform processing according to the present invention, and vice versa.

Although the present invention has been described above with reference to specific details such as specific components and limited examples and drawings, these have been provided only to help a more general understanding of the present invention, and the present invention is not limited to the above examples, and those with common knowledge in the technical field to which the present invention pertains may make various modifications and changes based on this description.

Therefore, the idea of the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are included in the scope of the idea of the present invention.

100: Communication network
200: Standardization System
210: Data Tokenization
220: Similarity calculation section
230: Data Standardization Department
240: Communications Department
250: Control Unit
300: Device

Claims (15)

A method implemented in a standardization system for standardizing text data on food materials, wherein the system includes a data tokenization unit, a similarity calculation unit and a data standardization unit,
The above data tokenization unit is a step of tokenizing text data regarding food materials into multiple tokens.
The step of the similarity calculation unit determining the weight of each of the plurality of tokens using an artificial intelligence-based token weight determination model, and calculating the food ingredient name similarity, which is the similarity between each of the plurality of standard food ingredient names included in the standard food ingredient data and the food ingredient name included in the text data, based on the weight, and
The above data standardization unit includes a step of standardizing text data regarding the food ingredient by determining a target food ingredient name corresponding to the text data among the plurality of standard food ingredient names based on the similarity of the food ingredient names,
The above token weight determination model is learned based on ontology data on food materials,
In the above calculation step, the similarity calculation unit further calculates the unit similarity, which is the similarity between the unit information of the food ingredient included in the text data and the unit information corresponding to each standard food ingredient name,
In the above standardization step, the data standardization unit determines the target food ingredient name based on the unit similarity,
In the above calculation step, the similarity calculation unit calculates the unit similarity based on the edit distance between the unit information of the food ingredient and the unit information corresponding to each standard food ingredient name.
In the above standardization step, the data standardization unit determines the target food ingredient name based on the weights assigned to each of the food ingredient name similarity and the unit similarity.
method. delete In the first paragraph,
The learning data of the above token weight determination model is augmented by hard negative sampling based on the above ontology data.
method. In the first paragraph,
In the above calculation step, the similarity calculation unit calculates the food ingredient name similarity such that the influence of the noisy token among the plurality of tokens on the food ingredient name similarity is smaller than the influence of the non-noisy token on the food ingredient name similarity.
method. delete delete In the first paragraph,
In the above calculation step, the similarity calculation unit calculates the unit similarity such that the influence of the noisy token among the plurality of tokens on the unit similarity is smaller than the influence of the non-noisy token on the unit similarity.
method. A non-transitory computer-readable recording medium recording a computer program for executing the method according to claim 1. As a system for standardizing text data on food materials,
Data tokenization unit that tokenizes text data about food ingredients into multiple tokens;
A similarity calculation unit that determines the weight of each of the plurality of tokens using an artificial intelligence-based token weight determination model, and calculates the food ingredient name similarity, which is the similarity between each of the plurality of standard food ingredient names included in the standard food ingredient data and the food ingredient name included in the text data, based on the weight; and
A data standardization unit is included to standardize text data regarding the food ingredient by determining a target food ingredient name corresponding to the text data among the plurality of standard food ingredient names based on the similarity of the food ingredient names.
The above token weight determination model is learned based on ontology data on food materials,
The above similarity calculation unit further calculates unit similarity, which is the similarity between the unit information of the food ingredient included in the text data and the unit information corresponding to each standard food ingredient name,
The above data standardization unit determines the target food ingredient name based further on the unit similarity,
The above similarity calculation unit calculates the unit similarity based on the edit distance between the unit information of the food ingredient and the unit information corresponding to each standard food ingredient name.
The above data standardization unit determines the target food ingredient name based on the weights assigned to each of the food ingredient name similarity and the unit similarity.
System. delete In Article 9,
The learning data of the above token weight determination model is augmented by hard negative sampling based on the above ontology data.
System. In Article 9,
The above similarity calculation unit calculates the food ingredient name similarity such that the influence of the noisy token among the plurality of tokens on the food ingredient name similarity is smaller than the influence of the non-noisy token on the food ingredient name similarity.
System. delete delete In Article 9,
The above similarity calculation unit calculates the unit similarity so that the influence of the noisy token among the plurality of tokens on the unit similarity is smaller than the influence of the non-noisy token on the unit similarity.
System.

Images