KR102763200B1 - Apparatus and method for extracting domain adaptive data - Google Patents

Abstract

The present disclosure relates to a domain adaptive data extraction device and method, comprising: a communication module for performing communication with an external device; a memory storing at least one process for performing a domain adaptive data extraction operation; and a processor for performing the domain adaptive data extraction operation according to the process, wherein the processor defines at least one basic keyword corresponding to each of a first domain and a second domain, collects words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one basic keyword, calculates word distribution entropy for each keyword in a first data pool to extract at least one core keyword, and extracts at least one data according to a preset condition based on the at least one core keyword in a second data pool, wherein the first domain represents a domain to which a model that a user wants to create is to be applied, and the second domain represents at least one domain that a user wants to exclude from the first domain, and there may be at least one or more of them.

Description

{APPARATUS AND METHOD FOR EXTRACTING DOMAIN ADAPTIVE DATA}

The present disclosure relates to a data extraction device and method, and more particularly, to a domain adaptive data extraction device and method.

In order to create a model for a specific domain, it is most important to secure not just a large amount of data, but also the appropriate data. However, the amount of refined data is increasing day by day, but it is very difficult to find out which data is appropriate for a specific domain.

In the first place, it is not common to store metadata that indicates what kind of information and characteristics the data contains together with each data, and even if the data appears to be similar, if the situation or purpose is different, it is often not a perfect fit in reality. In other words, even in the same financial domain, the terms and expressions used by Bank A and Insurance Company B may be different, and even within the same Bank A, the situations for consultation and internal meetings are different. Accordingly, usually, the data is reprocessed and labeled to fit the new domain, which is an extremely expensive task.

Also, when creating an artificial intelligence (AI) model for a specific domain, it is a task to define what that 'domain' is. Specifically, it is not easy to define how to define it and what data should be included, and even if you try to create new data, you have to define it first, but there is a dilemma that it is not easy in itself.

Therefore, there is a need to develop a technology that can define a domain using only a few keywords and easily extract meaningful data related to it.

Korean Patent Publication No. 10-2580512 (Registration Date: September 15, 2023)

The present disclosure provides a domain-adaptive data extraction device and method that can define a target domain and surrounding domains based on only a few keywords through a hybrid approach that uses both keyword and semantic search, thereby making it easier to extract not only data with similar texts but also data with similar contexts.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

According to an aspect of the present disclosure for achieving the above-described technical problem, a domain adaptive data extraction device comprises: a communication module for performing communication with an external device; a memory storing at least one process for performing a domain adaptive data extraction operation; and a processor for performing the domain adaptive data extraction operation according to the process, wherein the processor defines at least one basic keyword corresponding to each of a first domain and a second domain, collects words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one basic keyword, calculates word distribution entropy for each keyword in a first data pool to extract at least one core keyword, and extracts at least one data according to a preset condition based on the at least one core keyword in a second data pool, wherein the first domain represents a domain to which a model that a user wants to create is to be applied, and the second domain represents at least one domain that a user wants to exclude from the first domain, and there may be at least one or more of them.

Meanwhile, a domain adaptive data extraction method according to one aspect of the present disclosure includes: a step of collecting words excluding stop words as at least one keyword based on the at least one basic keyword corresponding to each of a first domain and a second domain; a step of extracting at least one core keyword by calculating word distribution entropy for each keyword within a first data pool; and a step of extracting at least one data according to a preset condition based on the at least one core keyword within a second data pool, wherein the first domain represents a domain to which a model that a user wants to create is to be applied, and the second domain represents at least one domain that a user wants to exclude from the first domain, and may be at least one or more.

In addition, a computer program stored in a computer-readable recording medium for executing a method for implementing the present disclosure may be further provided.

In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the aforementioned problem solving means of the present disclosure, a hybrid method using both keywords and semantic search can be used to define a target domain and surrounding domains based on only a few keywords, thereby making it easier to extract data with similar contexts as well as data with similar texts.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

FIG. 1 is a diagram illustrating a network structure of a domain adaptive data extraction system according to one embodiment of the present disclosure.
FIG. 2 is a diagram showing the configuration of a service server for domain adaptive data extraction according to one embodiment of the present disclosure.
FIG. 3 is a diagram illustrating a domain adaptive data extraction method according to one embodiment of the present disclosure.
FIG. 4 is a diagram illustrating an operation of collecting at least one keyword for each domain in a domain adaptive data extraction method according to one embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an operation of extracting at least one core data based on at least one keyword in a domain adaptive data extraction method according to one embodiment of the present disclosure.
FIG. 6 is a diagram illustrating an operation of extracting similar data for each of at least one data in a domain adaptive data extraction method according to one embodiment of the present disclosure.

The advantages and features of the present disclosure, and the methods for achieving them, will become apparent by referring to the embodiments described in detail below together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms, and these embodiments are provided only to make the present disclosure complete and to fully inform a person skilled in the art of the scope of the present disclosure, and the present disclosure is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present disclosure. In the present disclosure, the singular also includes the plural unless the context clearly dictates otherwise. The terms “comprises” and/or “comprising” as used herein do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the specification, and “and/or” includes each and every combination of one or more of the mentioned components. Although “first”, “second”, etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present disclosure.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with the meaning commonly understood by those skilled in the art to which this disclosure belongs. In addition, terms defined in commonly used dictionaries shall not be ideally or excessively interpreted unless explicitly specifically defined.

Throughout this disclosure, the same reference numerals refer to the same components. This disclosure does not describe all elements of the embodiments, and general contents or overlapping contents between embodiments in the technical field to which this disclosure belongs are omitted. The term "part" or "module" used in the specification means a software, hardware component such as an FPGA or an ASIC, and the "part" or "module" performs certain roles. However, the "part" or "module" is not limited to software or hardware. The "part" or "module" may be configured to be in an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the "part" or "module" includes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and “sub-components” or “modules” may be combined into a smaller number of components and “sub-components” or “modules” or further separated into additional components and “sub-components” or “modules”.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only a direct connection but also an indirect connection, and an indirect connection includes a connection via a wireless communications network.

Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

Throughout the specification, when it is said that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where there is another element between the two elements.

The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Singular expressions include plural expressions unless the context clearly indicates otherwise.

The identification codes in each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

The terms used in the following explanation are defined as follows.

Although described in this specification as a 'service server', this is a device for providing a domain adaptive data extraction service and may include various devices capable of performing computational processing. In addition, this service server may be connected/interconnected with a separate server, computer, and/or portable terminal to change settings or collect or analyze information and provide it, and its type and form are not limited.

Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The above server is a server that processes information by communicating with external devices, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

The above portable terminal may include, for example, all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted-device (HMD).

In this specification, 'domain' refers to a document composed of words/text.

The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

FIG. 1 is a diagram illustrating a network structure of a domain adaptive data extraction system according to one embodiment of the present disclosure.

Referring to FIG. 1, a domain adaptive data extraction system (hereinafter, referred to as “data extraction system”) (10) according to one embodiment of the present disclosure may be configured to include at least one of a service server (100) and a user terminal (200).

The service server (100) is intended to provide a domain-adaptive data extraction service (hereinafter referred to as a “data extraction service”) to a user, and for this purpose, a separate web page and/or platform (application) may be provided, and the user terminal (200) may provide or receive various information/data for the data extraction service based on the web page or platform.

When a user defines a keyword for each of the first domain and the second domain by executing a separate web page or platform through a user terminal (200), the service server (100) expands the defined keyword and then acquires at least one data based on the keyword. At this time, the service server (100) performs crawling to expand the defined keyword, acquires various data (information, content, etc.) through a preset website (300), and can expand the keyword using nouns included in the data.

Next, the service server (100) extracts only data related to the first domain by extracting core keywords from at least one data. Here, the first domain (target domain) represents a domain to which a model that the user wants to create will be applied, and the second domain (surrounding domain) represents at least one domain that the user wants to exclude from the first domain, and there may be at least one or more.

Accordingly, the service server (100) can provide the user with a data set composed of more suitable data by extracting not only data with similar texts but also data with similar context (meaning) using only a small number of keywords input by the user.

Meanwhile, the user terminal (200) is a terminal possessed by a user who wishes to receive a data extraction service through the service server (100). In order to receive the data extraction service, the user terminal may access a web page provided by the service server (100) or install a platform (application).

Accordingly, when the user terminal (200) executes a separate web page or platform to request service provision from the service server (100) to create a model of a specific domain, the user terminal (200) receives a data set composed of data extracted through the service server (100) in response thereto.

At this time, the user terminal (200) can request service provision by entering and defining keywords for each of the first domain and the second domain.

At this time, the user terminal (200) may be a computer, an Ultra Mobile PC (UMPC), a workstation, a netbook, a PDA (Personal Digital Assistants), a portable computer, a web tablet, a wireless phone, a mobile phone, a smart phone, a pad, a smart watch, a wearable terminal, an e-book, a portable multimedia player (PMP), a portable game console, a navigation device, a black box or a digital camera, or other mobile communication terminal, on which a user can install and execute a plurality of desired application programs (i.e., applications). That is, the user terminal (200) may be provided in various forms, and its form is not limited.

Although only one user terminal (200) is illustrated in FIG. 1, this is only for convenience of explanation, and there may be at least one or more, and the number and type are not limited.

As described above, the data extraction system according to the present disclosure can be implemented through data/information transmission and reception between a network-based service server (100) and a user terminal (200).

FIG. 2 is a diagram showing the configuration of a service server for domain adaptive data extraction according to one embodiment of the present disclosure.

Referring to FIG. 2, a service server (100) according to one embodiment of the present disclosure may be configured to include at least one of a communication module (110), a memory (120), and a processor (130).

The communication module (110) can communicate with at least one of various terminals (devices), external storage (e.g., database (140)), external server, and cloud server.

Meanwhile, an external server or a cloud server may be configured to perform at least a part of the role of the processor (130). That is, data processing or data operations, etc. may be performed on an external server or a cloud server, and the present disclosure does not place any special restrictions on this method.

Meanwhile, the communication module (110) can support various communication methods according to the communication standards of the communicating target (e.g., electronic device, external server, device, etc.).

For example, the communication module (110) may be configured to communicate with a communication target using at least one of the following technologies: WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), Infrared Data Association; IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus).

Meanwhile, the memory (120) may be configured to store various information related to the present disclosure. In the present disclosure, the memory (120) may be provided in the device itself according to the present disclosure. Alternatively, at least a part of the memory (120) may mean at least one of a database (DB, 140) and a cloud storage (or a cloud server). That is, it may be understood that the memory (120) is sufficient as a space where information required for the device and method according to the present disclosure is stored, and there is no limitation on the physical space. Accordingly, in the following, the memory (120), the database (140), the external storage, and the cloud storage (or the cloud server) are not separately distinguished, and are all expressed as the memory (120).

This memory (120) can store a plurality of application programs (or applications) driven by the service server (100), data for the operation of the service server (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. Meanwhile, the application programs can be stored in at least one memory provided in the memory (120), installed on the service server (100), and driven to perform operations (or functions) by at least one processor stored in the memory (120) via the processor (130).

Meanwhile, at least one memory may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, an SD or XD memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read-Only Memory (ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Programmable Read-Only Memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory may store information temporarily, permanently, or semi-permanently, and may be provided as a built-in or removable type.

Next, the processor (130) may be configured to control the overall operation of the device related to the present disclosure. The processor (130) may process signals, data, information, etc. input or output through the components discussed above, or provide or process appropriate information or functions to the user.

The processor (130) includes at least one CPU (Central Processing Unit) and can perform functions according to the present disclosure.

Specifically, the processor (130) defines at least one keyword corresponding to each of the first domain and the second domain, collects words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one keyword, calculates word distribution entropy for each keyword within the first data pool to extract at least one core keyword, and extracts at least one data according to a preset condition based on the extracted at least one core keyword within the second data pool.

At this time, at least one method or at least one search engine may be applied (used) to collect at least one keyword. For example, at least one keyword may be collected using a crawling method. However, this is only one embodiment and is not limited to this method.

Furthermore, the processor (130) can additionally extract contextually similar data for each of at least one data previously extracted within the second data pool. As a result, the data set can be supplemented by further extracting at least one data and its similar data within the second data pool.

Here, the preset condition may indicate the frequency of occurrence of a core keyword included in one piece of data. However, this is only an example, and conditions that satisfy regular expressions (regex), conditions for keywords to be excluded, etc. may be applied, and are not limited thereto. In addition, the first data pool may be a data pool in which at least one collected keyword is stored, and the second data pool may be a data pool in which various types of data are stored.

Meanwhile, the processor (130) extracts core keywords by calculating word distribution entropy for each of at least one keyword collected based on a preset algorithm, and extracts only data including the core keywords.

In addition, when the processor (130) collects at least one keyword, it searches each of the first domain and the second domain within the preset website (300), and performs crawling based on data within a preset number of contents located at the top among the contents identified as the search results. Thereafter, the processor (130) extracts words excluding stop words from the crawled text to generate a word list. At this time, an analysis method such as morphological analysis may be applied to extract words excluding stop words.

Meanwhile, when extracting core keywords, the processor (130) generates a document-term matrix (DTM) based on at least one keyword collected for each of the first domain and the second domain, and calculates word distribution entropy for each of at least one keyword based on a preset algorithm to determine weights. Thereafter, the processor (130) applies weights to each keyword included in the document-term matrix to calculate a score, normalizes each row of the document-term matrix, and extracts a preset number of keywords having relatively high scores as at least one core keyword.

Here, the preset algorithm is an algorithm that modifies TF-IDF (Term Frequency-Inverse Document Frequency), and is intended to calculate a score according to the importance of at least one keyword by the processor (130). Since it is different from the case where TD-IDF is generally used, it is difficult to calculate a meaningful value if it is used as is, so a modified algorithm is used. In other words, since the number of documents means the number of domains, the number of documents is much smaller than the case where TD-IDF is generally used, while the number of words in one word is much larger. Accordingly, TF is applied much more excessively than IDF, so entropy is used and normalization is performed. On the other hand, although the definition of DF itself indicates the number of documents in which the word appears at least once, since the number of words in one document (domain) is very large, it is a value that does not reflect the situation well to determine whether there is even one word, so word distribution entropy, not whether a word appears, is used as DF.

Specifically, TF means how frequently a keyword appears in a specific document (data). In other words, if a keyword appears frequently in a document, it is highly likely that it is related to the topic of the document, and thus it can be recognized as an important keyword. In addition, IDF means how frequently a keyword appears in the entire document (data). In other words, a keyword that appears frequently in the entire document can be recognized as an unimportant keyword. In the present invention, by utilizing these characteristics, keywords that appear frequently in each data are first identified based on TF, and if it is confirmed based on IDF that the keyword appears frequently in all data, the keyword is considered an unimportant keyword. In addition, keywords that appear frequently in each data are identified based on TF, and if it is confirmed based on IDF that the keyword appears frequently only in specific data among the entire data, the keyword is considered an important keyword for the specific data.

At this time, the IDF value can be calculated by taking the reciprocal of the value (entropy value) calculated from the entropy of each keyword. That is, the lower the entropy value, in other words, the higher the reciprocal value, the higher the keyword is used in specific documents rather than in various documents, and as a result, the TF-IDF value becomes higher.

Afterwards, the IDF value is multiplied for each keyword in the document word matrix to produce a TD-IDF score, and each row of the matrix is normalized. At this time, the normalization method may use at least one of Standard normalization, Min-Max normalization, Z-score normalization, MaxAbs normalization, Decimal Scaling normalization, power transformation, robust normalization, and log transformation. However, considering the characteristic that it may be sensitive to outliers, it may be configured to use robust normalization or log transformation among them.

In addition, the processor (130) may be configured to use either robust scaling or log transformation as a normalization method, taking into account the characteristic that it may be sensitive to outliers, but this is only one embodiment and does not limit the type of the method.

Meanwhile, when extracting data from within the second data pool, the processor (130) can obtain at least one data by checking the frequency of occurrence of each core keyword previously extracted for each data included in the second data pool and comparing the checked frequency of occurrence with a preset threshold. In other words, only data having a checked frequency of occurrence higher than the preset threshold can be selected and obtained.

In addition, the specific operation of the processor (130) will be described below based on each drawing.

At least one component may be added or deleted in response to the performance of the components illustrated in FIG. 2. For example, although not illustrated in FIG. 1, a search module, an augmentation module, and an extraction module may be configured to be controlled by a processor (130), respectively. In addition, it will be readily understood by those skilled in the art that the mutual positions of the components may be changed in response to the performance or structure of the device.

FIG. 3 is a diagram illustrating a domain adaptive data extraction method according to one embodiment of the present disclosure.

Referring to FIG. 3, when at least one basic keyword corresponding to each of the first domain and the second domain is input and defined from the user terminal (200), the service server (100) collects words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one basic keyword (S110).

Next, the service server (100) calculates word distribution entropy for each of at least one keyword collected by step S110 within the first data pool based on a preset algorithm to extract at least one core keyword (S120).

Next, the service server (100) extracts at least one data according to preset conditions based on at least one core keyword extracted by step S120 within the second data pool (S130).

In this way, a data set is created using the extracted data (S140).

Meanwhile, although not shown in FIG. 3, when performing step S140 or after performing step S140, contextually similar data may be additionally extracted for each of at least one data extracted by step S130 within the second data pool to reinforce the data set.

FIG. 4 is a diagram showing an operation of collecting at least one keyword for each domain in a domain adaptive data extraction method according to one embodiment of the present disclosure, and illustrates step S110 of FIG. 3 in more detail.

Referring to FIG. 4, the service server (100) searches each of the first domain and the second domain within a preset website (300) (S111), and performs crawling based on data within a preset number of contents located at the top among the contents identified as the search results (S112).

Next, the service server (100) extracts words excluding stop words from the text crawled through data by step S112 as at least one keyword (S113), and creates a word list using the at least one extracted keyword (S114).

FIG. 5 is a diagram illustrating an operation of extracting at least one core keyword based on at least one keyword in a domain adaptive data extraction method according to one embodiment of the present disclosure, and illustrates step S120 of FIG. 3 in more detail.

Referring to FIG. 5, the service server (100) checks the frequency of occurrence of each of at least one keyword collected for each of the first domain and the second domain by step S110 for each data included in the second data pool (S121), and generates a document-term matrix (DTM) using the checked frequency of occurrence (S122).

Next, the service server (100) calculates the word distribution entropy for each of at least one keyword collected by step S110 and determines a weight (S124).

Here, the word distribution entropy for each of at least one keyword collected can be expressed as in <Mathematical Formula 1> below.

Here, is a keyword The entropy of, is a domain Keywords within The frequency of occurrence (i.e., ) same as represents the number of the first domain and the second domain.

Next, the service server (100) calculates a score by applying the weight determined by step S123 to each keyword included in the document word matrix generated by step S122 (S124) and normalizes each row of the document word matrix (S125).

Here, the score for each keyword can be expressed as in <Mathematical Formula 2> below.

Here, is a domain Keywords for can indicate the frequency of appearance.

Accordingly, the corresponding keywords and their TF-IDF scores, i.e., can generate pairs of .

Accordingly, the service server (100) extracts a preset number of keywords with relatively high scores for each of at least one data as core keywords (S126).

FIG. 6 is a diagram illustrating an operation of extracting similar data for each of at least one data in a domain adaptive data extraction method according to one embodiment of the present disclosure, which is an additional operation that may be performed in step S140 of FIG. 3 or after step S140.

As described above, the service server (100) further extracts data having a similar embedding value similar to the embedding value of at least one data within the second data pool.

Referring to Fig. 6, in order to embed all data in the entire data pool, first, at least one data (data in the form of text including sentences, paragraphs, etc.) is passed through an embedding model to obtain a vector value as an embedding value of each data (S151), and data having an embedding value similar to the embedding value of each data is detected (S152). At this time, beyond simply extracting data having the same words, data having a similar context (meaning) can be found to diversify the data. Accordingly, by extracting not only at least one data but also data similar to each data, an enhanced data set can be generated.

The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should be referenced for additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with another computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with another computer or server located remotely using the communication module of the computer, what information or media to send and receive during communication, etc.

The above storage medium means a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device. That is, the program can be stored in various storage media on various servers that the computer can access or in various storage media on the user's computer. In addition, the medium can be distributed to computer systems connected to a network, so that a computer-readable code can be stored in a distributed manner.

The steps of a method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination of these. The software module may reside in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present disclosure pertains.

Although the embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present disclosure may be implemented in other specific forms without changing the technical spirit or essential features thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: Data extraction system 100: Service server
200: User terminal 300: Web page
110: Communication module 120: Memory
130: Processor 140: Database

Claims (10)

A communication module that performs communication with external devices;
A memory storing at least one process for performing domain adaptive data extraction operations; and
A processor for performing the domain adaptive data extraction operation according to the above process,
The above domain includes a first domain to which a model that the user wants to create is to be applied, and a second domain other than the first domain.
The above processor,
Define at least one basic keyword corresponding to each of the first domain and the second domain, and collect words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one basic keyword.
In the first data pool where the at least one keyword collected is stored, word distribution entropy for each of the at least one keyword collected is calculated to extract at least one core keyword,
Extracting at least one data according to a preset condition based on at least one core keyword from a second data pool where various types of data are stored,
A document-term matrix (DTM) is generated based on the at least one keyword for each of the first domain and the second domain, a word distribution entropy for each of the at least one keyword is calculated based on a preset algorithm to determine a weight, and a score is calculated by applying the weight to each keyword included in the document-term matrix, and then each row of the document-term matrix is normalized, and a preset number of keywords having relatively high scores are extracted as the at least one core keyword.
Characterized in that the normalization is performed based on one of the following methods: Standard normalization, Min-Max normalization, Z-score normalization, MaxAbs normalization, Decimal Scaling normalization, Power transformation, Robust scaling, and Log transformation.
Domain adaptive data extraction device. In the first paragraph,
The above processor,
When collecting at least one keyword, each of the first domain and the second domain is searched, crawling is performed based on a preset number of data within the content, and words excluding the stop words are extracted from the crawled text to create a word list.
Domain adaptive data extraction device. In the first paragraph,
The above conditions are set,
The frequency of occurrence of keywords contained in one piece of data,
The above processor,
When extracting at least one of the above data, the frequency of occurrence of each core keyword for each data included in the second data pool is checked, and only data having a confirmed frequency of occurrence higher than a preset threshold is selected and extracted.
Domain adaptive data extraction device. In the first paragraph,
The above processor,
Characterized in that it further extracts contextually similar data for each of the at least one data extracted within the second data pool.
Domain adaptive data extraction device. In paragraph 4,
The above processor,
Based on the above preset algorithm, the word distribution entropy for each keyword is calculated,
The above preset algorithm is,
It is an algorithm modified from TF-IDF (Term Frequency-Inverse Document Frequency), characterized in that it calculates a score according to the importance of at least one keyword.
Domain adaptive data extraction device. delete delete A communication module that performs communication with external devices;
A memory storing at least one process for performing domain adaptive data extraction operations; and
A processor for performing the domain adaptive data extraction operation according to the above process,
The above domain includes a first domain to which a model that the user wants to create is to be applied, and a second domain other than the first domain.
The above processor,
Define at least one basic keyword corresponding to each of the first domain and the second domain, and collect words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one basic keyword.
In the first data pool where the at least one keyword collected is stored, word distribution entropy for each of the at least one keyword collected is calculated to extract at least one core keyword,
Extracting at least one data according to a preset condition based on at least one core keyword from a second data pool where various types of data are stored,
The word distribution entropy for each of the above at least one keyword is,
It is characterized by being expressed as in the following <Mathematical Formula 1>.
Domain adaptive data extraction device.
<Mathematical formula 1>

Here, is a keyword The entropy of, is a domain Keywords within Frequency of appearance, represents the number of the first domain and the second domain. A communication module that performs communication with external devices;
A memory storing at least one process for performing domain adaptive data extraction operations; and
A processor for performing the domain adaptive data extraction operation according to the above process,
The above domain includes a first domain to which a model that the user wants to create is to be applied, and a second domain other than the first domain.
The above processor,
Define at least one basic keyword corresponding to each of the first domain and the second domain, and collect words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one basic keyword.
In the first data pool where the at least one keyword collected is stored, word distribution entropy for each of the at least one keyword collected is calculated to extract at least one core keyword,
Extracting at least one data according to a preset condition based on at least one core keyword from a second data pool where various types of data are stored,
The above processor,
A document-term matrix (DTM) is generated based on the at least one keyword for each of the first domain and the second domain, a word distribution entropy for each of the at least one keyword is calculated based on a preset algorithm to determine a weight, and a score is calculated by applying the weight to each keyword included in the document-term matrix, and then each row of the document-term matrix is normalized, and a preset number of keywords having relatively high scores are extracted as the at least one core keyword.
The above scores are,
It is characterized by being expressed as in <Mathematical Formula 2> below.
Domain adaptive data extraction device.
<Mathematical formula 2>

Here, is a domain Keywords for represents the appearance frequency of keyword j, and IFD _j represents the reciprocal of the entropy value of keyword j. A domain adaptive data extraction method performed by a processor of a device,
The above domain includes a first domain to which a model that the user wants to create is to be applied, and a second domain other than the first domain.
When at least one basic keyword corresponding to each of the first domain and the second domain is defined, a step of collecting words excluding stop words as at least one keyword for each of the first domain and the second domain based on the at least one basic keyword;
A step of extracting at least one core keyword by calculating word distribution entropy for each of the at least one keyword collected in the first data pool where the at least one keyword collected is stored; and
It includes a step of extracting at least one data according to a preset condition based on at least one core keyword in a second data pool where various types of data are stored,
The above processor,
A document-term matrix (DTM) is generated based on the at least one keyword for each of the first domain and the second domain, a word distribution entropy for each of the at least one keyword is calculated based on a preset algorithm to determine a weight, and a score is calculated by applying the weight to each keyword included in the document-term matrix, and then each row of the document-term matrix is normalized, and a preset number of keywords having relatively high scores are extracted as the at least one core keyword.
Characterized in that the normalization is performed based on one of the following methods: Standard normalization, Min-Max normalization, Z-score normalization, MaxAbs normalization, Decimal Scaling normalization, Power transformation, Robust scaling, and Log transformation.
Domain adaptive data extraction method.