KR102748662B1 - Apparatus for selecting data to train artificial intelligence and method for controlling the same - Google Patents

Abstract

The present invention relates to an artificial intelligence training data selection device and a control method thereof, wherein the artificial intelligence training data selection method may include a step of performing a preliminary analysis on user data and setting a quality verification condition including a verification target and a quality criterion using a predefined association rule; a step of measuring quality information of the user data by verifying data characteristics of the user data including standard characteristics and content-based characteristics based on a pre-generated data analysis algorithm and the quality verification conditions; and a step of selecting learning data effective for artificial intelligence training based on the data characteristics and the quality information.

Description

{APPARATUS FOR SELECTING DATA TO TRAIN ARTIFICIAL INTELLIGENCE AND METHOD FOR CONTROLLING THE SAME}

The present invention relates to an artificial intelligence training data selection device and a control method thereof.

Recently, as machine learning technologies such as deep learning are being actively introduced in the fields of artificial intelligence related to vision and speech, which were considered difficult to realize with conventional technologies, the format of training data for artificial intelligence is also changing, not only from structured data based on conventional databases to unstructured data including images, videos, audio, and text, and there is an increasing trend of utilizing multi-modal data that combines structured data and unstructured data.

In addition, in order to secure the performance of artificial intelligence algorithms based on machine learning, not only a certain amount of learning data for training the algorithm but also the quality of the learning data is emphasized. However, conventional methods for measuring the quality of data for artificial intelligence learning are limited to verification methods based on quality indicators for structured data. Therefore, they are mainly used to confirm the suitability and compliance rate of data specifications that must be followed in the data production process, such as the format of files constituting structured data and the quantity and distribution of classes or instances.

In addition, even if data that satisfies the data specifications and quantity criteria are used, quality issues from the perspective of data use, such as whether meaningful domain characteristics regarding the tasks assigned to the artificial intelligence to be implemented are reflected and whether the training of the artificial intelligence algorithm is effective, cannot be explained by the conventional standard suitability or compliance rate alone, and are transferred to the artificial intelligence implementation stage, causing an increase in time and cost due to repetitive quality verification and making it difficult to focus capabilities on the artificial intelligence implementation task.

However, in the case of unstructured data including audiovisual data, in addition to satisfying legal requirements such as copyright or source indication and inaccurate de-identification, it is essential to conduct content-based verification regarding whether the data reflects meaningful domain characteristics related to the task assigned to AI, as well as whether there are problems such as data duplication, insufficient or excessive labeling areas that hinder training of AI algorithms and achievement of target performance, whether labeling is inaccurate, and whether the image is out of focus. However, it is not easy for people to directly check with their eyes and ears, so it is not easy to view or listen to a large amount of data one by one. As a result, a sampling inspection method is used in the field to inspect a portion of the data, making it difficult to measure quality precisely.

In addition, conventional data quality measurement, evaluation, or verification tasks that depend on manpower are performed by a number of workers, and involve human error due to external factors that cannot be measured, such as the experience and subjectivity of each worker performing the tasks, and fatigue due to the difficulty and workload of the tasks. Therefore, it is difficult to maintain consistency between quality measurement standards and results, and it is difficult to secure accuracy and speed.

As a result, in terms of the production and use of data required for artificial intelligence training, the conventional method is limited to verification from the perspective of standard conformity due to time and cost issues in quantitative indicators, and it is difficult to quantify or objectively explain the content-based quality of a large amount of data. In addition, the selection of data suitable for artificial intelligence training and the composition of datasets, as well as decision-making for re-collecting and re-processing unsuitable data, still have to rely entirely on human experience and subjective judgment. Therefore, securing and improving the performance of artificial intelligence can be said to be a task that requires a lot of time, effort, and cost.

The background technology of this application is disclosed in Korean Patent Publication No. 10-0803561.

The present invention is intended to solve the problems of the above-mentioned prior art, and to provide an artificial intelligence training data selection device and a control method thereof, which enables easy setting of verification items and detailed quality indicators by pre-analyzing the composition and content-based characteristics of structured data collected or processed for training artificial intelligence algorithms as well as unstructured data of images, videos, audios, texts, or other types, and objectively measuring and evaluating quality using pre-generated data analysis algorithms for standard verification and content-based verification according to these settings, and providing the results in a form that can be visually explained and interacted with by the user, thereby enabling easy confirmation of the standard and content-based verification results of all or individual data together with statistics by quality indicator, and thereby facilitating the derivation of a dataset effective for artificial intelligence training by searching and selecting suitable or unsuitable data.

However, the technical tasks to be achieved by the embodiments of the present invention are not limited to the technical tasks described above, and other technical tasks may exist.

As a technical means for achieving the above technical task, a method for selecting artificial intelligence training data according to an embodiment of the present invention may include a step of performing a preliminary analysis on user data and setting a quality verification condition including a verification target and a quality criterion using a predefined association rule; a step of measuring quality information of the user data by verifying data characteristics of the user data including standard characteristics and content-based characteristics based on a pre-generated data analysis algorithm and the quality verification conditions; and a step of selecting learning data effective for artificial intelligence training based on the data characteristics and the quality information.

According to one embodiment of the present invention, the step of setting the quality verification condition may include the step of performing a preliminary analysis related to the type and format of the user data to generate standard information including configuration information and standard attribute information; the step of selecting a verification item including a detailed quality indicator using the standard information and the association rule, and determining a candidate for a valid standard value from the user data regarding the quality criterion to preliminarily set the quality verification condition; and the step of performing a detailed setting of the quality criterion based on a user input acquired through a user interface to set the quality criterion.

According to one embodiment of the present invention, the step of generating the specification information may include: identifying the specification (File format) of each file constituting the user data, generating the configuration information including classification information categorized according to the type and format of the data for each file specification and quantity information of the data; and extracting the specification attribute information defined as a list of valid values or a data schema for attributes for each specification based on the file specification, and generating the specification information combined with the configuration information.

According to one embodiment of the present invention, the step of pre-setting the quality verification conditions may include the step of generating a list of verification items and verification target data for at least one of verification of standard conformity verification and content-based verification based on the result of the pre-analysis and the association rule; and the step of selecting a list of indicator candidates based on the detailed quality indicators and determining preliminary quality information generated by using at least one of the standard attribute information, the list of valid values, and the data schema necessary for verification of each indicator from the standard attribute information to generate a quality criterion candidate value.

According to one embodiment of the present invention, the step of setting the quality criterion may determine the quality verification condition based on user input obtained through a user interface with respect to the prior quality information.

According to one embodiment of the present invention, the step of measuring the quality information of the user data may include the step of deriving quantitative quality information and criterion conformity information for each detailed index according to the standard conformity verification and content-based verification regarding the user data based on the data analysis algorithm that inputs the user data, the verification target, and the quality criterion; and the step of combining the statistical and categorization results regarding each quality characteristic with a list of associated user data by using the quantitative quality information and the criterion conformity information derived by extracting each detailed characteristic regarding the structural characteristics of the data determined by the pre-analysis or by generating or categorizing statistical information from the data analysis results of the content-based verification, thereby providing a quality verification result.

According to one embodiment of the present invention, the criterion suitability information includes information determining whether the user data satisfies the quality criterion, and can be determined based on whether a quantity or ratio of the suitable user data satisfies a target included in the quantitative quality information.

According to one embodiment of the present invention, the step of selecting the learning data may include the step of generating a visualization report on the structural characteristics of the user data based on the measurement results of the quality information and the criterion suitability information; and the step of performing a content-based data visualization search using the visualization report and the quality and characteristics of the user data as search conditions.

According to one embodiment of the present invention, the step of setting the quality verification condition includes the step of adding an algorithm and an association rule for performing verification of a quality indicator according to a format of the user data, and in the step of setting the quality verification condition and the step of selecting the learning data, the added algorithm and quality indicator information may be provided.

According to one embodiment of the present invention, the user data may include at least one of structured data, unstructured data, and label data, and may have at least one mixed form with respect to data type and format.

According to one embodiment of the present invention, the association rules may define the data analysis algorithms that can be performed with respect to predefined verification items for standard conformity verification and content-based verification, and the types and formats of the user data that can be processed with respect to the data analysis algorithms.

According to one embodiment of the present invention, the data analysis algorithm may perform at least one verification among the standard conformity verification and the content-based verification by inputting the user data and the quality criterion for each verification item defined in advance with respect to the standard conformity verification and the content-based verification, and provide the result as output.

According to one embodiment of the present invention, the predefined verification items may be defined as a list of detailed verification items or quality indicators that can be measured from the user data with respect to the standard conformity verification and the content-based verification, or as a functional specification of the data analysis algorithm that performs each verification.

An artificial intelligence training data selection device according to one embodiment of the present invention may include a quality verification condition setting unit that performs a preliminary analysis on user data and sets a quality verification condition including a verification target and a quality criterion using a predefined association rule; a data analysis and quality measurement unit that measures quality information of the user data by verifying data characteristics of the user data including standard characteristics and content-based characteristics based on a pre-generated data analysis algorithm and the quality verification conditions; and a learning data selection unit that selects learning data that is effective for artificial intelligence training based on the data characteristics and the quality information.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described means for solving the problem of the present invention, an artificial intelligence training data selection device and a control method thereof are provided, which adaptively sets verification targets and quality criteria through prior analysis with respect to input user data and performs standard conformity and content-based verification of user data based on a predefined data analysis algorithm, thereby providing an artificial intelligence training data selection device and method that enable easy searching and selection according to the purpose of use of the input user data using measured quality information and easily derive a valid dataset for artificial intelligence training.

According to the aforementioned means for solving the problem of this invention, an artificial intelligence training data selection device and its control method are provided that completely or partially automates the quality measurement work of data, which has been heavily dependent on manpower in the past, to minimize user intervention, thereby enabling mass comprehensive inspection while saving time and cost, and maintaining consistency in the method and criteria of quality measurement, thereby ensuring the objectivity of the verification result.

According to the above-described means for solving the problem of the present invention, by providing an artificial intelligence training data selection device and its control method for selecting unsuitable data based on quality measurement results in the data production process, there is an effect that can be utilized for objective and rapid decision-making regarding necessary measures for achieving data quality goals, such as re-collection or re-processing.

According to the above-described means for solving the problem of this invention, based on the results of quality measurement, a list of data or a quality verification report is provided as a result of selecting valid or invalid data according to the purpose of data use, thereby facilitating conversion of data formats for input of artificial intelligence algorithms, composition of conditional learning datasets according to the tasks of artificial intelligence, etc., thereby providing an artificial intelligence training data selection device and a control method thereof, thereby enabling time and effort to be concentrated on tasks of implementing application systems such as artificial intelligence algorithm training and performance testing.

According to the solution to the aforementioned problem of this invention, it is possible to introduce and utilize not only the model optimization method (model-centric) that is mainly used as a method for improving the performance of conventional artificial intelligence algorithms, but also the data-centric optimization method (data-centric), thereby supporting the advancement of artificial intelligence technology that users want to acquire from a wider range of perspectives.

According to the above-mentioned means for solving the problem of the present invention, by providing an artificial intelligence training data selection device and a control method thereof that facilitate the addition or change of a data analysis algorithm that implements a new quality indicator and its measurement method, there is an effect that it can be utilized for data analysis and quality verification having various conditions such as different data formats, contents, structures, and usage environments depending on the tasks assigned to artificial intelligence in the process of producing and using data.

However, the effects that can be obtained from this invention are not limited to the effects described above, and other effects may exist.

FIG. 1 is a schematic diagram of an artificial intelligence training data selection system including an artificial intelligence training data selection device according to one embodiment of the present invention.
Figure 2 is a drawing for explaining adaptively setting verification targets and quality criteria for quality verification conditions according to one embodiment of the present invention.
FIG. 3 is a schematic block diagram of an artificial intelligence training data selection device according to one embodiment of the present invention.
Figure 4 is a schematic diagram of an artificial intelligence training data selection device according to one embodiment of the present invention.
Figure 5 is a detailed configuration diagram of a quality verification condition setting unit according to one embodiment of the present invention.
Figure 6 is a detailed configuration diagram of a data analysis and quality measurement unit according to one embodiment of the present invention.
Figure 7 is a detailed configuration diagram of a learning data selection unit according to one embodiment of the present invention.
FIG. 8 is a diagram schematically illustrating semantic categorization of data types and formats according to one embodiment of the present invention.
FIG. 9 is a diagram showing a user interface for asking and answering preconditions in a pre-analysis process according to one embodiment of the present invention.
FIG. 10 is a drawing showing a work management screen that provides status and history information of standard verification and content-based verification tasks using a pre-generated data analysis algorithm according to one embodiment of the present invention.
Figure 11 is a drawing showing a verification result summary information provision screen that summarizes and provides quantitative quality information and standard conformity information derived from standard conformity and content-based verification according to one embodiment of the present invention.
Figure 12 is a drawing showing a verification result detailed information provision screen for viewing detailed results regarding standard conformity verification according to one embodiment of the present invention.
Figure 13 is a diagram for explaining the detailed results of content-based verification according to one embodiment of the present invention and the selection of a data set for artificial intelligence learning.
Figure 14 is an operational flow diagram of a control method of an artificial intelligence training data selection device according to one embodiment of the present invention.
FIG. 15 is a flowchart of an operation of a control method of an artificial intelligence training data selection device including another stepwise configuration according to one embodiment of the present invention.
FIG. 16 is a flowchart of an operation of a control method of an artificial intelligence training data selection device of another stepwise configuration according to one embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those with ordinary skill in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

Throughout this specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" or "indirectly connected" with another element in between.

Throughout this specification, when it is said that an element is located “on,” “above,” “below,” “under,” or “below” another element, this includes not only cases where an element is in contact with another element, but also cases where another element exists between the two elements.

Throughout this specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

FIG. 1 is a schematic diagram of an artificial intelligence training data selection system including an artificial intelligence training data selection device according to one embodiment of the present invention.

Referring to FIG. 1, the artificial intelligence training data selection system (10) of the present invention may include an artificial intelligence training data selection device (100) (hereinafter, also referred to as 'the present device') according to one embodiment of the present invention, an external server (200), and a user terminal (300). In addition, referring to FIG. 1, the artificial intelligence training data selection device (100) (hereinafter referred to as 'the device (100)') according to one embodiment of the present invention receives user data (A) collected or processed for the purpose of training an artificial intelligence algorithm from a data storage device such as a cloud storage or a network attached storage (NAS), as input, adaptively sets quality verification conditions (B) regarding verification targets and quality criteria based on a pre-analysis and pre-defined association rules regarding the input user data (A), performs data analysis and verification according to the standard conformity and content-based verification of the input user data (A) based on the quality verification conditions (B), and enables content-based data visualization search (D) that utilizes the quality and characteristics of visually expressed data regarding the user data (A) as search conditions based on the data verification results and statistical information (C) regarding the type of quality issue or distribution of data characteristics, thereby selecting and selecting learning data (E) suitable or valid for artificial intelligence training according to the purpose of data use and providing it to the user terminal (300). can operate. The external server (200) may include a data storage device such as cloud storage, network attached storage (NAS, Network Attached Storage), etc.

Here, the input user data (A) may be structured data stored in an external server such as a database, or unstructured data including images, videos, audio, text, etc., and may include label data composed of annotations such as ground truth assigned according to tasks that artificial intelligence must perform with respect to such raw data or source data, and may have one or more mixed forms with respect to such data types and formats, and may be composed of one or more files usable on a computer or storage device.

The present invention relates to a device and method for selecting artificial intelligence training data.

According to one embodiment of the present invention, by providing an artificial intelligence training data selection device and method, an existing human-based data quality verification method can be automated.

For example, when automating the existing human-based data quality verification method by an artificial intelligence training data selection device and method, the data subject to quality verification may be technically verifiable inspection items.

For example, in the past, problems such as blurring of image data, incorrect de-identification processing, and labeling area setting that does not form a single closed curve had to be directly checked by human eyes, but with the artificial intelligence training data selection device and method, the quality verification method for such problems can be automated.

Existing data quality verification is biased toward standard verification, so content-based quality issues are transferred to the AI development stage, requiring reinvestment of time and money to select valid data for AI training. In addition, it is impossible to inspect all data for AI training using human eyes and ears when a large amount of data is built, and verification criteria and results that depend on human experience and subjectivity may result in problems such as reduced objectivity.

However, according to the artificial intelligence training data selection device and method, it is possible to perform a comprehensive inspection on artificial intelligence learning data while maintaining objectivity.

According to one embodiment of the present invention, the objective verification indicators provided by the artificial intelligence training data selection device and method may include not only data specifications but also content-based verification, which is verification of items that previously had to be dependent on human eyes and ears.

According to one embodiment of the present invention, the artificial intelligence training data selection device and method can provide convenience to the user by presenting candidate option values that can be set through prior analysis, rather than leaving the setting of quality inspection options entirely to the user. For example, this can include designating inspection target items of labeling data for data standard and class distribution inspection.

According to one embodiment of the present invention, the AI training data selection device and method can search for data or select data related to desired characteristics using a visualized content-based search method based on the quality verification results. For example, according to the AI training data selection device and method, if a specific area such as a statistical graph or a heat map related to data characteristics is selected, a list of related data can be checked.

Since the existing verification method simply provides a quality inspection result sheet that only determines whether it conforms to the standard specification, the verification and selection of the data content ultimately depends on manual work. However, the artificial intelligence training data selection device and method can perform at least one of re-collection and re-processing of the data based on the verification result, or support rapid decision-making regarding the selection of available valid data.

As a result, the artificial intelligence training data selection device and method can focus time and capabilities on the artificial intelligence development work, which is the purpose of data construction.

According to one embodiment of the present invention, the above-described association rule used by the present device (100) may define the above-described generated data analysis algorithm that can be performed with respect to the above-described verification items for standard conformity verification and content-based verification, and the type and format of the above-described data that can be processed with respect to the above-described algorithm. In addition, according to one embodiment of the present invention, the above-described association rule may operate to enable not only the above-described rules but also the modification of existing rules or the addition of new rules in order to support the functional expansion of the present device (100).

In addition, the type and format of the user data (A) input to the device (100) according to one embodiment of the present invention may be classified into raw data, source data, and labeling data, and may be classified into file formats such as JPG, MP4, WAV, DCM, TXT, CSV, and JSON with respect to the standard for saving actual data as a file, and may be classified into image, video, audio, text, or other formats as the classification of information stored in the file format. However, the present invention is not limited thereto and may be defined by the association rules.

In addition, the above-mentioned pre-generated data analysis algorithm of the present device (100) according to one embodiment of the present invention may analyze and evaluate the standard or content-based characteristics of data by inputting the user data (A) and the quality verification condition (B) for each pre-defined verification item with respect to the standard conformity and content-based verification, and provide the results as output. For example, the above-mentioned pre-generated data analysis algorithm may be implemented through all types of data analysis and processing methods known in the past or to be developed in the future, such as regular expressions and formal languages, pattern recognition, audio and image signal processing, computer vision, rule-based, heuristic algorithms, decision trees, mathematical models and statistical methods, SVM (Support Vector Machine), artificial neural networks, reinforcement learning, natural language processing (NLP), and machine learning methods including deep learning.

In addition, the pre-defined verification items of the device (100) according to one embodiment of the present invention may be a list defining detailed verification items or quality indicators that can be measured from data with respect to the standard conformity verification and content-based verification, or may be defined as a functional specification of a pre-generated data analysis algorithm that performs each verification. In addition, according to one embodiment of the present invention, the device (100) may examine attributes related to data standards, such as file format, image resolution, and encoding format, according to the data type and format, to determine whether or not the data meets the standard conditions of a given quality criterion.

In addition, according to one embodiment of the present invention, the content-based verification method of the present device (100) can assist in making decisions easily by providing results analyzed based on the previously generated data analysis algorithm regarding quantifiable quality types among quality issues that were qualitatively evaluated based on the eyes and ears of humans in the past and, if necessary, domain knowledge. For example, when assuming that domain features included in image data as learning data for visual intelligence in terms of quantifiable quality types are originally expressed as adjacent pixel values, the present device (100) may include, but is not limited to, quality problem types such as, but not limited to, cases where focus blurs due to problems in the operation of photographing equipment or the environment, causing the corresponding features to be indistinguishable, or where a bounding box for an arbitrary pixel area included in an image as an annotation indicating a true value for an object classification problem may be set to be excessively small or large or biased toward a specific location, or where different labels are set in duplicate or the same area of an image, or where a polygon area set along the boundary of an object does not form a single closed curve, or where key points constituting a human skeleton are set outside the human body area and on the background, or where a labeling area is outside the image area, or where the image direction does not match, or where incomplete de-identification processing is set outside the human body area.

This device (100) can adaptively set quality verification conditions (B) regarding verification targets and quality criteria based on pre-analysis of input user data (A) and pre-defined association rules.

Specifically, the present device (100) can generate standard information on data types and formats by pre-analyzing user data (A), select applicable verification items using the standard information and pre-defined association rules, and provide preliminary quality information as output to preliminarily set verification conditions by adaptively determining candidates for valid reference values from the user data (A) regarding quality criteria.

In other words, before the user inputs verification conditions for verification items required for verification purposes with respect to the user's data (A), the device (100) intelligently generates a candidate list of valid setting values from the user data (A) and provides it in advance, thereby facilitating the setting of verification conditions even without knowing detailed data configuration information.

In addition, the present device (100) can pre-analyze user data (A) to generate the above-mentioned standard information and generate configuration information and standard attribute information.

According to one embodiment of the present invention, the configuration information of the user data (A) generated by the present device (100) may be determined by identifying the file format of each file constituting the input user data (A) and categorizing the classification information and the quantity information of the data according to the type and format of the data for each file format. In addition, according to one embodiment of the present invention, the present device (100) may determine the specification attribute information by using a list of valid values for each specification attribute or a data schema from the user data based on a unique specification attribute defined by a conventional technical standard or specification specification for the file format.

According to one embodiment of the present invention, the unique standard attribute of data may be determined by extracting a technical attribute that is uniquely defined in the related art for each data type and format using the configuration information of the user data (A). For example, it may be resolution, color format, compression format, etc. for images, and may be unique attributes such as bit rate, frame rate (Frames per Second, FPS) including some or all of the attributes of the image for videos, and may be unique attributes such as channel layout, sampling rate, etc. for audio, and may broadly include character encoding format, etc. for text. However, it is not limited thereto. In addition, according to one embodiment of the present invention, the device (100) may be determined by extracting header information or added metadata defined according to the storage structure of the corresponding file for the standard attribute, or may be determined as a confirmed value by interpreting the data.

In addition, according to one embodiment of the present invention, the data schema identified by the present device (100) is a data storage structure that arranges information units constituting data according to a certain rule, and a data structure that defines the format and input range of values stored with respect to the information units, and may be provided together with the input user data, or determined by interpreting actual data. For example, the present device (100) may check additional data provided together with the input user data (A), or may extract a syntactic structure by interpreting the user data (A) in various ways, such as a syntactic analysis method such as pattern recognition, regular expression, automata, and finite state machine, and a statistical method, and may be determined as information on the range of inputtable values, including the format and list of valid input values found in the data, and the input range of values that can be expressed as maximum and minimum values, or a formatted string.

In addition, the device (100) may, for preliminary setting of the verification conditions for the user data (A), generate a list of verification items and verification target data applicable to the user data among the standard conformity verification or content-based verification methods based on the standard information according to the predefined association rules, select a list of indicator candidates suitable for the user data among the detailed quality indicators included in the verification items, and adaptively set the pre-quality criteria in an adaptive manner using the list of standard attributes and valid values or data schema required for verification of each indicator from the standard attribute information of the pre-analysis results.

For example, the pre-quality criteria generated by the present device (100) may be determined by using the standard attribute information of the user data (A) to designate applicable items from a pre-defined verification item list for each data in a database or file unit, or by generating a data list corresponding to each of the pre-defined verification items, by using pre-defined association rules regarding the possibility of standard conformity verification or content-based verification for each type and format of the data, so as to facilitate decision-making and editing of verification conditions according to the user's final decision, and may be utilized to present quality criteria information expected to be suitable for the verification target by combining the standard characteristics representing the composition information of the data and the structural characteristics extracted from the user data (A) and the inherent characteristics of the data structure, and to display them through a user interface.

Figure 2 is a drawing for explaining adaptively setting verification targets and quality criteria for quality verification conditions according to one embodiment of the present invention.

Referring to FIG. 2, the device (100) according to one embodiment of the present invention preliminarily analyzes user data (A) for standard conformity and content-based verification, and generates preliminarily quality information (G) that preliminarily sets detailed quality indicators and a list of valid values for standard conformity and content-based verification based on the association rules (F) that define extractable data characteristics and verifiable item-by-item indicators according to data type and format. In addition, the device (100) according to one embodiment of the present invention provides preliminarily quality information (G) through a user interface that facilitates user review and editing, and can finally determine quality verification conditions (B) according to user decision-making.

For example, the pre-quality information (G) according to one embodiment of the present invention may be provided in the form of a choice (top of the example area of the user interface on the right side of Fig. 2) defined as a list of valid values for verification items and detailed indicators of the configuration information of the pre-analyzed user data (A), or may be provided in a visualized form (bottom of the example area of the user interface on the right side of Fig. 2) as a map, graph, table, etc. that facilitates selection of detailed data and setting of conditions for hierarchical syntax verification or statistical calculation regarding data distribution or characteristics required for content-based verification or using a data schema.

In addition, the quality verification condition (B) according to one embodiment of the present invention may not necessarily use only the values included in the pre-quality information (G) generated as a result of the pre-analysis. For example, the user may exclude or modify unnecessary inspection items and detailed indicators or valid values among the pre-quality information (G) presented in a given user interface, and may add values not found by the pre-analysis. In addition, the method may be determined in a way that is finally confirmed according to the user's decision by inputting the target quality criteria such as the verification item unit for determining whether each verification item complies with the criteria or the compliance rate or standard quantity for the detailed indicator of each verification item, as well as modifying the preset values.

According to one embodiment of the present invention, the device (100) can provide a quality verification condition setting menu, a data analysis and quality measurement menu, and a learning data selection menu to the user terminal (300). For example, the user terminal (300) can download and install an application program provided by the device (100), and the quality verification condition setting menu, the data analysis and quality measurement menu, and the learning data selection menu can be provided through the installed application.

This device (100) may include all types of servers, terminals, or devices that transmit and receive data, content, and various communication signals through a network with an external server (200) and a user terminal (300), and have the function of storing and processing data.

The user terminal (300) is a device that is connected to the main device (100) and the external server (200) through a network, and may be, for example, a smartphone, a smart pad, a tablet PC, a wearable device, and all types of wireless communication devices such as a PCS (Personal Communication System), a GSM (Global System for Mobile communication), a PDC (Personal Digital Cellular), a PHS (Personal Handyphone System), a PDA (Personal Digital Assistant), an IMT (International Mobile Telecommunication)-2000, a CDMA (Code Division Multiple Access)-2000, a W-CDMA (W-Code Division Multiple Access), a Wibro (Wireless Broadband Internet) terminal, and a fixed terminal such as a desktop computer or a smart TV.

Examples of networks for sharing information between the device (100), the external server (200), and the user terminal (300) may include, but are not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a 5G network, a WIMAX (World Interoperability for Microwave Access) network, a wired/wireless Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a Wifi network, an NFC (Near Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, etc.

According to one embodiment of the present invention, the device can perform pre-analysis on user data.

For example, the device can perform pre-analysis by analyzing the type and format of input user data to generate standard information.

According to one embodiment of the present invention, the device can set quality verification conditions including verification targets and quality criteria using predefined association rules.

For example, the device can select applicable verification methods and detailed quality indicators using standard information and predefined association rules, and adaptively determine candidates for valid reference values from user data regarding quality criteria.

According to one embodiment of the present invention, the device can measure quality information of user data by verifying data characteristics of user data including standard characteristics and content-based characteristics based on previously generated data analysis algorithms and quality verification conditions.

According to one embodiment of the present invention, the device can select learning data effective for artificial intelligence training based on data characteristics and quality information.

According to one embodiment of the present invention, the device can perform a preliminary analysis related to the type and format of user data to generate specification information including configuration information and specification attribute information.

According to one embodiment of the present invention, the device can select verification items including detailed quality indicators using standard information and association rules.

According to one embodiment of the present invention, the device can pre-set quality verification conditions by determining candidates for valid criteria from user data regarding quality criteria.

According to one embodiment of the present invention, the device can set quality criteria by performing detailed settings for quality criteria based on user input obtained through a user interface.

According to one embodiment of the present invention, the device can identify the file format of each file constituting user data and generate configuration information including classification information categorized according to the type and format of data regarding each file format and quantity information of the data.

According to one embodiment of the present invention, the device can extract specification attribute information defined as a list of valid values for each specification attribute or a data schema based on a file specification, and generate specification information combined with configuration information.

According to one embodiment of the present invention, the device can generate a list of verification items and verification target data for at least one of standard conformity verification and content-based verification based on the results of the pre-analysis and the association rules.

According to one embodiment of the present invention, the device can generate a quality criterion candidate value by determining a priori quality information generated by selecting a list of indicator candidates based on detailed quality indicators and using at least one of standard attribute information, a list of valid values, and a data schema necessary for verification of each indicator from standard attribute information.

According to one embodiment of the present invention, the device can determine quality verification conditions based on user input obtained through a user interface regarding prior quality information.

According to one embodiment of the present invention, the device can derive quantitative quality information and standard conformity information for each detailed index according to standard conformity verification and content-based verification regarding user data based on a data analysis algorithm that takes user data, a verification target, and a quality criterion as inputs.

According to one embodiment of the present invention, the device can provide a quality verification result by combining the statistical and categorization results on each quality characteristic with a list of associated user data using quantitative quality information and criterion suitability information determined by extracting each detailed characteristic on the structural characteristics of data determined by a pre-analysis or generating statistical information from the data analysis results of content-based verification or by categorizing and determining the statistical information.

According to one embodiment of the present invention, the criterion suitability information includes information determining whether user data conforms to a quality criterion, and may be determined based on whether a quantity or ratio of suitable user data satisfies a target included in the quantitative quality information.

According to one embodiment of the present invention, the device can generate a visualization report on the structural characteristics of user data based on the measurement results for the quality information and the criterion conformity information.

According to one embodiment of the present invention, the device can perform content-based data visualization search using the quality and characteristics of user data as search conditions using a visualization report.

According to one embodiment of the present invention, the device may add an algorithm and association rules for performing verification of quality indicators according to the format of user data.

According to one embodiment of the present invention, in the step of setting quality verification conditions and the step of selecting learning data, additional algorithm and quality indicator information may be provided.

According to one embodiment of the present invention, the user data may include at least one of structured data, unstructured data, and label data, and may have at least one mixed form with respect to data type and format.

According to one embodiment of the present invention, association rules may define data analysis algorithms that can be performed with respect to predefined verification items for conformance verification and content-based verification, and types and formats of user data that can be processed with respect to the data analysis algorithms.

According to one embodiment of the present invention, a data analysis algorithm may perform at least one verification among standard conformity verification and content-based verification by inputting user data and quality criteria for each verification item defined in advance for standard conformity verification and content-based verification, and provide the result as output.

According to one embodiment of the present invention, the predefined verification items may be defined as a list of detailed verification items or quality indicators that can be measured from user data with respect to standard conformity verification and content-based verification, or as a functional specification of a data analysis algorithm that performs each verification.

For example, an AI-based learning model may be created through AI-based learning such as machine learning or deep learning, but is not limited thereto, and various neural network systems or algorithms that have been developed or will be developed in the future may be applied.

Additionally, the present device can be implemented as follows.

A control method of an artificial intelligence training data selection device according to one embodiment of the present invention may include a quality verification condition setting step of pre-analyzing input user data and adaptively setting a verification target and quality criteria using pre-defined association rules, a data analysis and quality measurement step of objectively performing standard conformity and content-based verification of input user data according to quality verification conditions based on a pre-defined data analysis algorithm, and a learning data selection step of outputting and providing data valid for artificial intelligence training by searching and selecting in a visually explainable manner according to the purpose of use of the input user data based on objectively verified data characteristics and quality information.

In addition, the quality verification condition setting step may include a pre-analysis step for generating standard information by analyzing the type and format of input user data, and a verification condition pre-setting step for selecting applicable verification methods and detailed quality indicators using the standard information and pre-defined association rules, and adaptively determining candidates for valid reference values from user data regarding quality criteria.

In addition, the pre-analysis step identifies the file format of each file that constitutes the input user data, and generates data configuration information including classification information categorized according to the type and format of data for each file format and quantity information for the data, and based on unique specification attributes defined by conventional technical standards or specification specifications for the file format, data specification attribute information can be generated using a list of valid values for each specification attribute or a data schema from the user data.

In addition, the verification condition preliminary setting step may include a step of generating a list of verification items and verification target data applicable to user data among the standard conformity verification or content-based verification methods based on predefined association rules and preliminary analysis results, and a step of adaptively determining preliminary quality information that is preliminary generated by selecting a list of indicator candidates suitable for user data among the detailed quality indicators included in the verification items and using a list of standard attributes and valid values or a data schema required for verification of each indicator from standard attribute information.

In addition, the pre-quality information can be utilized to minimize the user's effort by facilitating the setting of quantitative quality goals regarding the quality criteria of data to be verified by referring to valid quality verification conditions applicable to user data.

In addition, the data analysis and quality measurement steps can objectively verify the entered user data according to the verification target and quality criteria to produce quantitative quality information and criteria suitability information.

Additionally, quality information and criterion conformity information can be determined by pre-generated data analysis algorithms that perform standard conformity and content-based verification.

Additionally, quality information may be determined by extracting each detailed characteristic regarding the structural characteristics of data determined by prior analysis by this device or by generating or categorizing statistical information from the results of content-based data analysis.

Additionally, the criterion suitability information may be quantitatively determined by determining whether the input user data conforms to the quality criterion and determining whether the quantity or proportion of the conforming data meets the quantitative quality goal.

In addition, a method for controlling an artificial intelligence training data selection device according to one embodiment of the present invention may include a step of generating a visualization report so that quality information and standard suitability information can be visually explained in relation to structural characteristics of data in order to select learning data effective for artificial intelligence training, and a content-based data visualization search step that utilizes the quality and characteristics of the visually expressed data as search conditions.

In addition, the method for controlling an artificial intelligence training data selection device according to one embodiment of the present invention can ensure functional expandability by adding a new algorithm and association rule that performs verification of a new user data format and a quality indicator suitable therefor as well as a previously generated data analysis algorithm and utilizing it to control the user data processing flow.

Meanwhile, an artificial intelligence training data selection device according to one embodiment of the present invention may include a quality verification condition setting unit that pre-analyzes input user data and adaptively sets verification targets and quality criteria using pre-defined association rules; a data analysis and quality measurement unit that objectively performs standard conformity and content-based verification of input user data according to quality verification conditions based on a pre-generated data analysis algorithm; and a learning data selection unit that searches and selects input user data in a visually explainable manner according to the purpose of use using measured quality information and outputs valid data for artificial intelligence training.

In addition, the quality verification condition setting unit may include a pre-analysis unit that analyzes the type and format of input user data to generate standard information, a verification condition preliminary setting unit that selects applicable verification methods and verification items regarding detailed quality indicators using the standard information and predefined association rules and adaptively determines candidates for valid standard values from user data regarding quality criteria, and a quality criteria setting unit that includes a user interface that enables detailed settings such as modification, addition, and deletion for the preliminary quality criteria.

In addition, the pre-analysis unit examines the input user data to identify the file format and generate classification information categorized by the type and format of the data and configuration information regarding quantity, and extracts specification attribute information defined as a list of valid values for each specification attribute or a data schema from the user data based on unique specification attributes defined by conventional technical standards or specification specifications, and generates specification information combined with the configuration information.

In addition, the verification condition preliminary setting unit may include a verification target identification unit that generates a list of verification items and verification target data applicable to user data among the standard conformity verification or content-based verification methods based on the preliminary analysis results, and a quality criterion candidate value generation unit that selects a list of indicator candidates suitable for user data among the detailed quality indicators included in the verification items, and adaptively sets preliminary quality information using a list of standard attributes and valid values or a data schema required for verification of each indicator from the preliminary analysis results.

In addition, the data analysis and quality measurement unit may include an analysis algorithm application unit that derives quantitative quality information and criterion conformity information for each detailed indicator according to standard conformity and content-based verification regarding user data based on a predefined data analysis algorithm that takes user data and verification target and quality criterion information as inputs, and a data verification result generation unit that extracts detailed characteristics regarding the structural characteristics of data and uses the derived quantitative quality information and criterion conformity information to provide quality verification results by combining statistical and categorization results regarding each quality characteristic with a list of associated user data.

In addition, the learning data selection unit may include a visualization report generation unit that generates a visually explainable report on the structural characteristics of data based on the quality verification results, a visualization query unit that utilizes the quality and characteristics of the visually expressed data as search conditions to facilitate content-based data visualization search in a user-interactive manner, a final judgment of suitability by the user, and selection of valid data, and a data output unit that outputs a list of data selected or a quality verification report according to the purpose of data use.

In addition, the artificial intelligence training data selection device according to one embodiment of the present invention does not operate by being limited to a fixed list of available algorithms, considering that user data may have various types and formats depending on the task assigned to the artificial intelligence, but may include an algorithm management unit that adds or facilitates the use of new algorithms that perform verification of new user data formats and quality indicators suitable therefor as well as predefined data analysis algorithms, and a task management unit that controls the data processing flow by providing algorithm and quality indicator information to the pre-analysis unit and the training data selection unit.

FIG. 3 is a schematic block diagram of an artificial intelligence training data selection device according to one embodiment of the present invention.

Referring to FIG. 3, the artificial intelligence training data selection device may include a quality verification condition setting unit, a data analysis and quality measurement unit, and a learning data selection unit.

According to one embodiment of the present invention, the quality verification condition setting unit can perform a preliminary analysis on user data and set a quality verification condition including a verification target and quality criteria using predefined association rules.

According to one embodiment of the present invention, the data analysis and quality measurement unit can measure quality information of user data by verifying data characteristics of user data including standard characteristics and content-based characteristics based on previously generated data analysis algorithms and quality verification conditions.

According to one embodiment of the present invention, the learning data selection unit can select learning data effective for artificial intelligence training based on data characteristics and quality information.

Figure 4 is a schematic diagram of an artificial intelligence training data selection device according to one embodiment of the present invention.

Referring to FIG. 4, the device (100) may include a pre-analysis unit (110), a data analysis and quality measurement unit (120), a learning data selection unit (130), an algorithm management unit (140), and a task management unit (150).

The quality verification condition setting unit (110) can adaptively set verification targets and quality criteria using pre-analysis results and association rules regarding input user data (A).

The data analysis and quality measurement unit (120) can perform standard conformity and content-based verification of user data (A) according to the verification target and quality criteria set above based on the previously generated data analysis algorithm.

The learning data selection unit (130) can use the measured quality information to search and select user data (A) according to the purpose of use, thereby deriving a valid data set for artificial intelligence training.

The algorithm management unit (140) can add a new user data format and a new algorithm that verifies quality indicators suitable for the new user data format, or register and manage association rules (F) regarding the application method of the new data format and the algorithm, taking into account the functional expandability of the device (100).

The work management unit (150) can control the data processing flow of the quality verification condition setting unit (110), the data analysis and quality measurement unit (120), and the learning data selection unit (130) by using the management information regarding the algorithm and association rules (F) of the algorithm management unit (140).

In addition, according to one embodiment of the present invention, the device (100) can add a new algorithm that performs verification of a new user data format and a quality indicator suitable therefor as well as an existing data analysis algorithm, considering that user data may have various types and formats depending on the task assigned to the artificial intelligence, and can ensure functional expandability by adding an association rule (F) regarding a method of applying the new data format and algorithm and utilizing it to control the user data processing flow.

Figure 5 is a detailed configuration diagram of a quality verification condition setting unit according to one embodiment of the present invention.

Referring to FIG. 5, the quality verification condition setting unit (110) may include a pre-analysis unit (111), a verification condition preliminary setting unit (112), and a quality standard setting unit (113).

The pre-analysis unit (111) can analyze the type and format of user data (A) using association rules (F) to generate standard information including data composition information and standard attribute information.

The verification condition preliminary setting unit (112) may include a verification target identification unit (112-1) and a quality criterion candidate value generation unit (112-2).

In addition, the verification condition pre-setting unit (112) according to one embodiment of the present invention can generate the pre-quality information (G) in an adaptive manner by using the standard information and association rules (F) analyzed by the pre-analysis unit (111). For example, the verification target identification unit (112-1) can distinguish data requiring standard verification and data to be subject to content-based data analysis and verification according to association rules, and the quality criterion candidate value generation unit (112-2) can define valid verification items and detailed indicators for each verification target, and extract and combine the associated data structure and attribute information from the user data (A) based on the analyzed standard information for each quality criterion to provide the pre-quality information (G) including the candidate list of valid values for the quality criterion as an output.

The quality standard setting unit (113) can provide the user with detailed settings such as modification, addition, and deletion of the above-mentioned pre-quality information (G) to finally confirm the quality verification conditions (B).

Figure 6 is a detailed configuration diagram of a data analysis and quality measurement unit according to one embodiment of the present invention.

Referring to FIG. 6, the data analysis and quality measurement unit (120) may include an analysis algorithm application unit (121) and a data verification result generation unit (122).

The algorithm application unit (121) can derive and provide quantitative quality information and standard conformity information for each detailed indicator according to standard conformity and content-based verification of user data based on a pre-generated data analysis algorithm that uses user data (A) and quality verification conditions (B) as input.

The data verification result generation unit (122) extracts detailed characteristics regarding the structural characteristics of data from the algorithm application unit (121), derives a quality verification result by combining the statistical and categorization results regarding each quality characteristic with the file list of related user data using quantitative quality information and criterion suitability information derived from the algorithm application unit (121), and provides a content-based data visualization search (D) that utilizes the quality and characteristics of visually expressed data regarding user data (A) as search conditions based on the data verification result and statistical information (C) regarding the type of quality problem or distribution of data characteristics.

Figure 7 is a detailed configuration diagram of a learning data selection unit according to one embodiment of the present invention.

Referring to FIG. 7, the learning data selection unit (130) may include a visualization report generation unit (131), a visualization query unit (132), and a data output unit (133).

The visualization report generation unit (131) can provide visualized quality information that allows user interaction and data list information for each characteristic, with respect to data characteristics measured by quality indicators, along with suitability according to quality criteria, using the quality verification results for user data (A). For example, according to one embodiment of the present invention, the device (100) can output and provide content-based data visualization search (D) that utilizes the quality and characteristics of visually expressed data regarding user data (A) as search conditions based on the data verification results and statistical information (C) regarding the type of quality issue or distribution of data characteristics.

The visualization query unit (132) can facilitate the review and change of the appropriate criteria for the search and quality of related data according to detailed quality indicators and data characteristics using content-based data visualization search (D), thereby allowing the user to finalize the quality of data and selection of valid data.

The data output unit (133) can provide a list of data or a quality verification report as a result of selecting valid or invalid data according to the purpose of data use based on the quality measurement results.

The algorithm management unit (140) adds or facilitates the use of new algorithms that perform verification of new user data formats and quality indicators suitable therefor, as well as previously generated data analysis algorithms. In other words, the present device (100) according to one embodiment of the present invention does not operate by being limited to a fixed list of available algorithms, considering that user data (A) may have various types and formats depending on the task assigned to the artificial intelligence.

The work management unit (150) may include a work management unit that provides algorithm and quality indicator information to the pre-analysis unit and the learning data selection unit to control the data processing flow.

FIG. 8 is a diagram schematically illustrating semantic categorization of data types and formats according to one embodiment of the present invention.

Referring to FIG. 8, the artificial intelligence training data selection device can determine a data type according to a file format of the data, perform data classification based on the data type, and set the classified data as user data.

FIG. 9 is a diagram showing a user interface for asking and answering preconditions in a pre-analysis process according to one embodiment of the present invention.

Referring to FIG. 9, the device (100) according to one embodiment of the present invention may receive a precondition (H) for applying an association rule (F) from a user using a predetermined user interface while performing a pre-analysis. According to one embodiment of the present invention, in the process of generating pre-quality information (G) using the association rule (F), the pre-quality information may be generated only using the association rule (F), or may be adaptively generated by applying the association rule (F) by referring to the precondition (H) input according to a question-and-answer session with a user using a predetermined user interface, taking into account user data configured in various ways depending on the purpose of use.

For example, the input of a precondition (H) using a given user interface may relate to a user selection for determining a file format of data estimated as labeling data during a pre-analysis, or may relate to information on the format of a label, such as a bounding box or polygon, and the location and format of annotations related to coordinate values expressing the label, depending on the type of artificial intelligence task to be implemented using the data. In addition, the present device (100) according to one embodiment of the present invention may operate in a manner such as extracting a data schema according to the input precondition (H), or estimating the input range of coordinate data and the median of the area distribution to present them as a reference value candidate for setting the minimum label size, but is not limited thereto, and may adaptively analyze and generate data in a manner that requires user intervention and confirmation regarding the precondition (H) necessary for controlling the selection and operation of an algorithm that implements a data analysis and verification method.

In addition, the device (100) according to one embodiment of the present invention can store and retain in a database the precondition (H) information together with the prequality information (G) or the quality verification condition (B) information that has been reviewed and modified by the user and finalized. Accordingly, the user can change the precondition (H) at any time to perform the pre-analysis again, or can provide the user with the ability to quickly complete the user settings and start the quality verification by reusing the prequality information (G) or the finalized quality verification condition (B) information that includes the same precondition (H) for new user data additionally generated for the same purpose.

FIG. 10 is a drawing showing a work management screen that provides status and history information of a standard conformity and content-based verification task using a pre-generated data analysis algorithm according to one embodiment of the present invention.

Referring to FIG. 10, the device (100) can derive data verification results and statistical information (C) according to the standard conformity and content-based verification of the input user data (A) based on the pre-generated data analysis algorithm using the quality verification condition (B). In addition, referring to FIG. 10, the work management screen provided by the device (100) may be provided with a work status display unit (1001) including the status of a verification task regarding the user data (A) and a work registration unit (1002) in which new data to be verified can be input. For example, the work status display unit (1001) may include information such as the status of the task indicating whether the task is waiting, completed, stopped, or the progress, the name of the data input by the user, the quantity and main data format of the data being inspected, the inspection date, and the like, and may include a user interface that calls a result inquiry unit (1003) in which the progress of the task in progress or the result of the task can be viewed.

The data quantity and main data format displayed on the task status display unit (1001) according to one embodiment of the present invention can be generated through the aforementioned prior analysis. For example, the configuration information regarding the data format constituting the input user data (A) can be determined by examining each file format from the entire file list regarding the user data (A) to generate a categorized list regarding the file format, recording the number of times a file of the corresponding format occurs, and selecting the format with the highest occurrence frequency. In addition, a predefined icon or image regarding the file format with the highest occurrence frequency can be displayed on the task status display unit (1001) so that it is easy to visually distinguish which format of data each task mainly has, but the information and the display method displayed on the task status display unit (1001) are not limited thereto.

In addition, the work management screen provided by the device (100) according to one embodiment of the present invention can check and modify the verification target and quality criteria assigned to each data displayed on the work status display unit (1001), restart verification with changed conditions, select start and stop of work, delete verification history, or register new data using the work registration unit (1002) while duplicating preconditions (H) and quality verification conditions (B) set for other data, or set and apply new verification conditions through prior analysis. In addition, according to one embodiment of the present invention, the registration and verification history for user data (A) can be stored and retained in a database and provided so that the user can check it. At this time, pre-quality information (G) including the precondition (H) or confirmed quality verification condition (B) information can be stored together so that the change history of verification conditions for user data (A) can be managed.

Figure 11 is a drawing showing a verification result summary information provision screen that summarizes and provides quantitative quality information and standard conformity information derived from standard conformity and content-based verification according to one embodiment of the present invention.

Referring to FIG. 11, the verification result summary information provision screen provided by the present device (100) may include a summary result display section (1004) according to each verification of standard conformity verification and content-based verification items along with inspection overview information such as the name and quantity of the verification target data, and may include a user interface for displaying a detailed result display section (1005) by selecting detailed items of the summary result. For example, the present device (100) may analyze user data (A) entered according to the quality verification conditions (B) using a previously generated data analysis algorithm to produce quantitative quality information and standard conformity information and provide a summary result and a detailed result.

According to one embodiment of the present invention, the quality information provided by the present device (100) may be determined by extracting each detailed characteristic regarding the structural characteristics of the data determined by the above-described pre-analysis and generating or categorizing statistical information, and may be used to provide a quality verification result by combining it with a list of associated user data (A). In addition, according to one embodiment of the present invention, the criterion suitability information provided by the present device (100) may be determined by determining suitability or unsuitability according to whether the quality information complies with a quality criterion included in the quality verification condition (B).

According to one embodiment of the present invention, the verification result summary information provided by the present device (100) can indicate suitability by using the target quality criteria for the inspection items as quantitative goals in addition to the quality criteria for each detailed quality index input by the user. For example, the suitability is determined based on the quality criteria set for each detailed quality index for the input user data (A), and the verification result for each detailed quality index is generated for each inspection item, and statistics on the quantity of user data files that violate the quality criteria are calculated and whether the ratio or quantity has achieved the target quality criteria, i.e., the quantitative goal, of the inspection items, thereby determining suitability.

In addition, referring to FIG. 11, whether the quantitative goal of each inspection item is achieved or not may be displayed not only as a ratio or quantity of suitable or unsuitable data, but also in a visualization format such as a bar graph or a pie chart. For example, the graph may display a quantitative goal set by a user as a dotted line, etc., as shown in the bar graph of FIG. 11, and the suitable and unsuitable ratios may be displayed in different colors. In addition, depending on the embodiment, the summary result display section for each verification item may include a user interface such as a button or link for calling or displaying detailed results.

Figure 12 is a drawing showing a verification result detailed information provision screen for viewing detailed results regarding standard conformity verification according to one embodiment of the present invention.

Referring to FIG. 12, the verification result detailed information provision screen provided by the present device (100) may include a detailed result display section (1005) that provides a verification overview, such as the result of the judgment of conformity with respect to the quantitative target for each verification item for the standard conformity verification, the quantity of verification target data, the quantity of data with quality issues, etc., and the detailed verification result of the detailed quality indicator for the verification item, in a visualized form of a graph. For example, the detailed result display section (1005) may display the ratio of data with quality issues for each detailed quality indicator or display them sorted by occurrence frequency so that the user can easily identify what quality issue he or she should pay attention to, and may display the result of the conformity or unconformity judgment together, but is not limited thereto and may be expressed in various ways (such as (1006) or (1007) of FIG. 13) depending on the characteristics of the data or the verification item.

In addition, referring to FIG. 12, the list view button included in the detailed result display unit (1005) provided by the present device (100) or the individual items corresponding to each quality indicator visually expressed as bars or pies of each graph or chart can be selected to search for a data list (I) for each quality issue type from the user data list. For example, if verification has been performed on image data to which keypoint labels have been assigned, the searched data list (I) can be searched for in detail by searching or sorting according to the type and suitability of the quality issue or the format and name of the file, and the source data and labeling data can be cross-referenced and searched for as a content-based data visualization search (D), such as providing a preview by overlapping keypoint labels on image data from the data list (I).

Figure 13 is a diagram for explaining the detailed results of content-based verification according to one embodiment of the present invention and the selection of a data set for artificial intelligence learning.

Referring to FIG. 13, the device (100) uses a data visualization display unit (1007) that provides various statistical visualization methods including a structural data display unit (1006), so that quality information and standard suitability information of the quality verification result regarding the user data (A) can be visually explained in terms of the structural characteristics of the data, and a content-based data visualization search (D) method that utilizes the quality and characteristics of the visually expressed data as search conditions can be used to retrieve or select the user data (A) and derive and provide a valid data set (E) for artificial intelligence training.

In addition, in order to enable content-based data visualization search (D) provided by the device (100) according to one embodiment of the present invention, standard conformity and content-based verification results may be input to visually convert display target items and maintain a list of related data for each item. For example, the detailed result display unit (1005) may use the data visualization display unit (1007) to visualize verification results by quality issue type and provide a list of data (I) regarding related quality issues when selecting each visualization element, or may use a face detection algorithm as a previously generated data analysis algorithm to detect a person's face in an image and visualize statistical information regarding whether or not a face is detected for each data, thereby utilizing the content-based data visualization search (D) to confirm and search data characteristics and select a list of data (I).

According to one embodiment of the present invention, the data visualization display unit (1007) provided by the present device (100) can provide data in a two-dimensional or three-dimensional format so that the structural characteristics of the data can be easily confirmed, including statistical values, with respect to the content-based data analysis and verification results. For example, the structural data display unit (1006) can display the hierarchical structure of the data, such as the syntactic structure of the labeling data, so that the quality verification results regarding the conformity of each lower element to the standard can be confirmed, and visualization methods such as histograms, scatter plots, and box plots can be combined to display the distribution of data instances regarding annotations or the correlation between data characteristics.

In addition, according to one embodiment of the present invention, the visualization information provided using the data visualization display unit (1007) can be utilized as a search tool by using the visualized information from the content-based verification result derived by referring to the original data and the labeling data. For example, by using the object labeling area information, the sharpness of the labeling area from the original data, the image data, can be investigated, and the sharpness distribution of the entire data can be displayed using a histogram, or the threshold values (cut-off values) for the minimum and maximum values can be adjusted so that only the data within the range can be selected, or a heat map can be generated with respect to the location and size of the labeling area defined by a bounding box or a polygon, or a partial area of the visualized data can be selected according to various visualization methods, such as a bar of a histogram or a grid of a heat map, and a detailed data list (I) included in the range of the characteristics or values indicated by the corresponding area can be output, or a content-based data visualization search (D) can be performed.

In addition, according to one embodiment of the present invention, the device (100) can perform detailed inquiry on the content-based characteristics of the user data (A) by using the content-based data visualization search (D) that utilizes the data visualization display unit (1007) in various combinations. For example, in the case of an artificial intelligence model that classifies the hand movements of a person in an image, when keypoint or joint information on a hand including fingers is required as the main learning data, the user may select an annotation on the hand from the structural data display unit (1006), search for a list of data that satisfies the standard criteria such as the number of keypoints and the range of coordinate values of the hand part, and use the data visualization display unit (1007) to check various quality indicators and data properties from the list, and select valid data based on the position or size of the hand, or select data instances based on the gender or age group of the actor performing the hand movement. However, the present invention is not limited to these utilization examples, and various utilization examples may exist depending on the characteristics and purpose of use of the data.

In addition, according to one embodiment of the present invention, the device (100) may output a list (I) of valid data selected based on a quantified quality scale of quality information measured by the user to derive a learning data set (E) or a report on the results of quality verification. For example, based on the report on the results of quality verification provided by the device (100) according to one embodiment of the present invention, it may be utilized for decision-making such as determining a list of target files to confirm the composition and quality status of data and determine required supplementary work regarding collection or processing, or it may be utilized for converting the format for inputting user data (A) into an artificial intelligence algorithm based on the list (I) of valid data selected using the data visualization display unit (1007) based on the file format of the original data, converting the syntax structure of the labeling data into a conventionally known format such as MS COCO or Pascal VOC, or dividing data for cross-verification of the artificial intelligence algorithm.

In addition, according to one embodiment of the present invention, the learning data set (E) for artificial intelligence training provided by the present device (100) may be generated by dividing it into data sets for cross-validation. For example, the selected valid data list (I) may be divided into learning data sets, verification data sets, and test data sets for cross-validation according to training of the artificial intelligence algorithm, or may be merged or divided and provided according to a ratio defined by the user.

In this way, the present device (100) performs standard conformity verification and content-based data analysis and verification using a pre-generated data analysis algorithm according to objectively defined quality verification conditions (B) with respect to a large amount of user data (A), and provides a content-based data visualization search (D) that utilizes the quality and characteristics of visually expressed data regarding the user data (A) as search conditions based on the data verification results and statistical information (C) regarding the type of quality issue or distribution of data characteristics from the results, thereby enabling a list of valid data (I) regarding each type of quality issue or data characteristic to be searched and selected according to the purpose of data use, or a learning data set (E) for artificial intelligence training to be configured, thereby reducing the effort of individually checking a large amount of data such as audiovisual data with one's eyes and ears, and making it easy to secure data necessary for artificial intelligence training using objective quality indicators or quickly perform decisions for data re-collection or re-processing, thereby improving the work efficiency of an artificial intelligence project.

Below, we will briefly review the operating flow of this system based on the detailed explanation above.

Figures 14 to 16 are flowcharts of operations for a control method of an artificial intelligence training data selection device according to one embodiment of the present invention.

The control method of the artificial intelligence training data selection device illustrated in FIGS. 14 to 16 can be performed by the artificial intelligence training data selection device described above. Therefore, even if the content is omitted below, the content described for the artificial intelligence training data selection device can be equally applied to the description of the control method of the artificial intelligence training data selection device.

Figure 14 is an operational flow diagram of a control method of an artificial intelligence training data selection device according to one embodiment of the present invention.

Referring to FIG. 14, a control method of an artificial intelligence training data selection device (also referred to as an 'artificial intelligence training data selection method') may include steps S110 to S130.

At step S110, the device can perform a pre-analysis on user data and set quality verification conditions including verification targets and quality criteria using predefined association rules.

Next, in step S120, the device can measure quality information of user data by verifying data characteristics of user data including standard characteristics and content-based characteristics based on previously generated data analysis algorithms and quality verification conditions.

Next, at step S130, the device can select learning data valid for artificial intelligence training based on data characteristics and quality information.

FIG. 15 is a flowchart of an artificial intelligence training data selection method using explainable quality information including another stepwise configuration according to one embodiment of the present invention.

Referring to FIG. 15, in step S210, the quality verification condition setting unit (110) can generate standard information for the input user data (A) and adaptively generate pre-quality information (G) regarding the quality verification condition (B) using the pre-defined association rule (F). In this step or before this step, the user data (A) can be input into the device (100), and the generated pre-quality information (G) can be provided through a user interface that facilitates review and modification by the user, and can provide the final quality verification condition (B) as an output according to the user's judgment.

Specifically, step S210 may include step S211 of generating standard information by analyzing the type and format of the input user data (A), step S212 of preliminary setting verification conditions by using the standard information to define candidates for applicable detailed quality indicators and valid reference values, and step S213 of reviewing or modifying and confirming the quality verification conditions (B). In this regard, in step S211, the standard analysis unit (111) may generate standard information according to the type and format of the input user data (A) of the input user data, and in step S212, the verification condition preliminary setting unit (112) may identify a verification target from the input user data (A) and adaptively generate a candidate list of valid values that can be designated as quality criteria with respect to applicable data analysis and verification conditions, and in step S213, the quality standard setting unit (113) may provide the user with detailed settings such as modification, addition, and deletion for the verification target and the candidate list of initially set valid values, thereby finally confirming the quality verification conditions (B).

Next, in step S220, the data analysis and quality measurement unit (120) can generate quantitative quality information and standard conformity information according to the quality verification condition (B) for the input user data (A). Here, the previously generated data analysis algorithm may mean an algorithm that analyzes and evaluates the standard or content-based characteristics of the data as input by the user data (A) and the quality criterion for each verification item defined in advance with respect to standard conformity and content-based verification, and provides the results as output.

Next, in step S230, the learning data selection unit (130) can select data according to the purpose of verification or use of the input user data (A) in a manner provided by the visualization report generation unit (131) and the visualization query unit (132) based on the quantified quality scale of the measured quality information.

Next, in step S240, the data output unit (133) included in the learning data selection unit (130) can output a list of data or a quality verification report as a result of selecting valid or invalid data according to the purpose of data use based on the quality measurement results.

FIG. 16 is a flowchart of an operation of a control method of an artificial intelligence training data selection device of another stepwise configuration according to one embodiment of the present invention.

Referring to FIG. 16, a control method of an artificial intelligence training data selection device can be performed as in the flowchart disclosed in FIG. 16.

In the above description, steps S110 to S130 and S210 to S240 may be further divided into additional steps or combined into fewer steps, depending on the implementation example of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be changed.

The control method of the artificial intelligence training data selection device according to one embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known to and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The above hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the control method of the artificial intelligence training data selection device described above can also be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present invention is for illustrative purposes only, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined manner.

The scope of the present application is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present application.

10: Artificial Intelligence Training Data Selection System
20: Network
100: Artificial Intelligence Training Data Selection Device
110: Quality verification condition setting section
111: Pre-analysis department
112: Verification Condition Preliminary Setting Section
112-1: Verification target identification section
112-2: Quality Criteria Candidate Value Generation Section
113: Quality Standards Setting Department
120: Data Analysis and Quality Measurement Department
121: Application of Analysis Algorithm
122: Data verification result generation section
130: Learning data selection section
131: Visualization Report Generation Section
132: Visualization query section
133: Data output section
140: Algorithm Management Department
150: Work Management Department
200: External server
300: User Terminal
1001: Work status indicator
1002: Work Register
1003: Results Inquiry
1004: Summary Results Display Section
1005: Detailed results display section
1006: Structured Data Display
1007: Data visualization display section
A: User data
B: Quality verification conditions
C: Data verification results and statistical information
D: Content-based data visualization search
E: Valid training data
F: Association Rules
G: Pre-quality information
H: Prerequisite
I: List of selected data

Claims (15)

In the method of selecting artificial intelligence training data,
A step of performing a preliminary analysis on user data and setting quality verification conditions including verification targets and quality criteria using predefined association rules;
A step of measuring the quality information of the user data by verifying the data characteristics of the user data including standard characteristics and content-based characteristics based on the generated data analysis algorithm and the quality verification conditions; and
A step of selecting learning data valid for artificial intelligence training based on the above data characteristics and the above quality information;
Including,
The step of setting the above quality verification conditions is:
A step of generating specification information including configuration information and specification attribute information by performing a preliminary analysis related to the type and format of the above user data;
A step of selecting verification items including detailed quality indicators using the above standard information and the above association rules, and determining candidates for valid reference values from the user data regarding the quality criteria to pre-set the quality verification conditions; and
A step of setting the quality criterion by performing detailed settings for the quality criterion based on user input obtained through a user interface; including;
The steps for generating the above standard information are:
A step for identifying the file format of each file composing the user data and generating the configuration information including classification information categorized according to the type and format of the data for each file specification and quantity information of the data; a step for extracting the specification attribute information defined as a list of valid values or a data schema for each specification attribute based on the file specification; and generating the specification information in which the specification attribute information and the configuration information are combined.
A method for selecting artificial intelligence training data. delete delete In the first paragraph,
The step of pre-setting the above quality verification conditions is:
A step of generating a list of verification items and verification target data for at least one of verification of conformity to the standard and content-based verification based on the results of the above pre-analysis and the above association rules; and
A step of selecting a list of indicator candidates based on the above detailed quality indicators and generating quality criterion candidate values by determining pre-quality information generated by using at least one of the above standard attribute information, the list of valid values, and the data schema necessary for verifying each indicator from the above standard attribute information.
which includes,
A method for selecting artificial intelligence training data. In paragraph 4,
The steps for setting the above quality criteria are:
The quality verification conditions are determined based on user input obtained through the user interface regarding the above-mentioned prior quality information.
A method for selecting artificial intelligence training data. In the first paragraph,
The step of measuring the quality information of the above user data is:
A step of deriving quantitative quality information and standard conformity information for each detailed indicator according to standard conformity verification and content-based verification of the user data based on the data analysis algorithm that uses the user data, the verification target, and the quality criterion as inputs; and
A step of extracting each detailed characteristic of the structural characteristics of the data determined by the above-mentioned preliminary analysis, or generating statistical information from the data analysis results of the content-based verification, or using the quantitative quality information and the above-mentioned criterion suitability information determined by categorizing, to provide a quality verification result by combining the statistical and categorization results of each quality characteristic with a list of associated user data.
which includes,
A method for selecting artificial intelligence training data. In Article 6,
The above criteria suitability information is:
Including information that determines whether the above user data satisfies the above quality criteria, and wherein the quantity or ratio of the above user data that satisfies the target included in the quantitative quality information is determined based on whether the quantity or ratio of the above user data that satisfies the target included in the quantitative quality information.
A method for selecting artificial intelligence training data. In Article 6,
The step of selecting the above learning data is:
A step of generating a visualization report on the structural characteristics of the user data based on the measurement results for the quality information and the criterion suitability information; and
A step of performing a content-based data visualization search using the quality and characteristics of the user data as search conditions using the above visualization report.
which includes,
A method for selecting artificial intelligence training data. In the first paragraph,
The step of setting the above quality verification conditions is:
A step of adding an algorithm and association rules for performing verification of quality indicators according to the format of the above user data;
Including,
In the step of setting the above quality verification conditions and the step of selecting the above learning data, the added algorithm and quality indicator information are provided.
A method for selecting artificial intelligence training data. In the first paragraph,
The above user data is,
Contains at least one of structured data, unstructured data and label data, and has at least one mixed form with respect to data type and format.
A method for selecting artificial intelligence training data. In the first paragraph,
The above association rules are,
Defining the data analysis algorithm that can be performed on the predefined verification items for standard conformity verification and content-based verification, and the types and formats of the user data that can be processed with respect to the data analysis algorithm.
A method for selecting artificial intelligence training data. In the first paragraph,
The above data analysis algorithm is,
The result of performing at least one of the standard conformity verification and the content-based verification by inputting the user data and the quality criteria for each verification item defined in relation to the standard conformity verification and the content-based verification is provided as an output.
A method for selecting artificial intelligence training data. In Article 12,
The above defined verification items are:
A list defining detailed verification items or quality indicators that can be measured from the user data with respect to the above standard conformity verification and the above content-based verification, or defined as a functional specification of the data analysis algorithm that performs each verification.
A method for selecting artificial intelligence training data. In the artificial intelligence training data selection device,
A quality verification condition setting unit that performs a preliminary analysis on user data and sets quality verification conditions including verification targets and quality criteria using predefined association rules;
A data analysis and quality measurement unit that measures the quality information of the user data by verifying the data characteristics of the user data including standard characteristics and content-based characteristics based on the generated data analysis algorithm and the quality verification conditions; and
A learning data selection unit that selects learning data that is valid for artificial intelligence training based on the above data characteristics and the above quality information;
Including,
The above quality verification condition setting unit includes a pre-analysis unit that analyzes the type and format of input user data to generate standard information, a verification condition preliminary setting unit that selects applicable verification methods and verification items for detailed quality indicators using standard information and predefined association rules and adaptively determines candidates for valid standard values from user data regarding quality criteria, and a quality criteria setting unit that includes a user interface that enables detailed settings such as modification, addition, and deletion for the pre-set quality criteria.
The pre-analysis unit examines the input user data to identify the file format and generate classification information categorized by the type and format of the data and configuration information on the quantity, and extracts a list of valid values for each attribute of the standard from the user data based on the unique standard attribute defined by the conventional technical standard or standard specification or the standard attribute information defined by the data schema to generate standard information combined with the configuration information.
Artificial intelligence training data selection device.
A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 1 and 4 to 13 on a computer.

Images