JP7631042B2 - Learning Data Generation System - Google Patents

Description

Embodiments of the present invention relate to learning data technology used in AI, machine learning, and deep learning.

AI can perform highly accurate analysis by performing segmentation on large amounts of related image data. Segmentation (Image Segmentation is the process of identifying regions of an image that are likely to be meaningful.

JP 2019-101535 A

We provide a learning data generation system that can efficiently and quickly segment image data used in learning models such as AI.

The training data generation system of the embodiment includes a color similarity calculation unit that calculates color similarity between image data obtained by photographing a training subject based on a color feature distribution of the image data; a pattern classification unit that classifies the image data into a plurality of patterns using the color similarity; a color adjustment unit that corrects the color tone of the image data using color adjustment parameters that are set for each pattern classification taking into account the shooting environment; a conversion unit that converts the color-tone corrected image data into grayscale; and an area designation unit that partitions areas in the grayscale converted image data and designates the areas based on a luminance threshold value that is set for each pattern classification.

FIG. 2 is an explanatory diagram of a segmentation function by the training data generation device of the first embodiment. FIG. 2 is a functional block diagram of the training data generating device according to the first embodiment. 10A to 10C are diagrams for explaining a method of calculating color similarity and pattern classification using color similarity in the first function of the first embodiment. 13A and 13B are diagrams for explaining area designation in the second function of the first embodiment. FIG. 11 is a diagram illustrating a processing flow of a learning data generation process according to the first embodiment.

The following describes the embodiment with reference to the drawings.

First Embodiment
1 to 5 are diagrams for explaining a training data generating device 100 according to a first embodiment, and FIG. 1 is a diagram for explaining a segmentation function of this embodiment.

The training data generation device 100 of this embodiment generates training data used for AI, machine learning, deep learning, and the like. The training data generation device 100 performs a segmentation process to determine where something is in image data, regardless of whether or not there is a teacher label (answer), and to identify areas in the image that are likely to be meaningful. By using the segmented training data as teacher data, the accuracy of AI, machine learning, and deep learning is improved.

However, segmentation cannot be performed automatically, and the range and area must be specified manually for each image data. For example, when creating training data, a person must manually identify the target part in the image and perform operations such as encircling the target area with dots. In particular, objects whose shapes change, such as clouds and bubbles, require detailed bordering. For this reason, segmentation processing requires a huge amount of time and human effort.

As mentioned above, large amounts of training data are required to improve the accuracy of AI, machine learning, and deep learning. However, generating training data (training data) with the segmentation required for highly accurate learning requires a huge amount of time and human effort, and as a result, the supply of training data cannot keep up with demand, resulting in the problem of AI, machine learning, deep learning, and other such technologies taking a long time to "learn."

As an example, in water treatment work, there is a stage where purification is carried out using microorganisms, and managing the state of the microorganisms is extremely important. Chemicals and air are added to activate the microorganisms, but the area, amount, color, and shape of the bubbles on the water surface change depending on the state of the microorganisms, and it is necessary to control the addition of chemicals and air depending on these changes. It is possible to analyze these with AI and control them automatically, but because a lot of time is spent on image segmentation processing, it is not possible to prepare enough training data necessary for learning. As a result, there have been cases where the learning process was so time-consuming that it was necessary to give up on analyzing with AI and controlling it automatically.

In addition, the shooting environment, i.e., the lighting environment including natural and/or artificial light, has a large influence on the segmentation process being performed manually. When the lighting environment changes, the appearance (color) of the learning object changes, the contours become unclear, and it becomes difficult to distinguish the learning object from other objects. For example, the same object looks different in an image where the sunlight is strong and the object appears white overall, and an image where the sunlight is blocked and the object appears dark overall. In addition, when the area, quantity, color, shape, etc. of the object change, it becomes even more difficult to automate the segmentation process. In other words, the segmentation process, which is necessary to improve accuracy, tends to increase the reliance on human region identification and region designation.

The training data generation device 100 of this embodiment automates the segmentation process and generates highly accurate training data quickly and in a short period of time.

As shown in FIG. 1, the learning data generation process of this embodiment is broadly composed of two functions. The first function is to analyze the color feature amounts of each captured image data to calculate a color feature distribution, and to calculate color similarity by comparing the color feature distributions between image data. Then, pattern classification is performed to group images with similar color similarity.

The second function performs color adjustment processing according to the shooting environment for each group of pattern-classified images. The images that have been color-adjusted are then converted to grayscale, and areas within the image data are partitioned and designated using a luminance threshold (brightness threshold) that is set for each group of pattern-classified images.

The first function classifies patterns using similarities based on the color feature distribution of each image, making it possible to group the appearance (color tone) of the learning subject into the same similar group, taking into account the influence of the shooting environment (lighting environment including natural light and/or artificial light). Therefore, processing appropriate for each pattern classification can be performed all at once.

The second function then performs color adjustments for each pattern classification in accordance with the shooting environment, while simultaneously performing grayscale conversion to identify a specific range of colors within the image (a color range that can serve as the area to be learned), and highlights the area to be specified using a luminance threshold (brightness threshold) set for each pattern classification.

In other words, the second function of this embodiment converts the image data into grayscale, narrowing down the color range that can be used as the learning target area based on the difference in brightness between colors, and for the narrowed down color range, highlighting the contours of the area range based on the brightness threshold. In this way, the contour of the learning target area in the image data is formed based on the brightness threshold, and area designation (area extraction) is performed by dividing the area into areas above the brightness threshold and areas below the brightness threshold, or into areas inside the contour and areas outside the contour.

This is done collectively for each group of pattern-classified images, so segmented image data, i.e., learning data, can be generated in large quantities in a short amount of time.

Figure 2 is a functional block diagram of the learning data generation device 100 of this embodiment. The imaging device captures an image that includes at least a part of the learning subject (monitoring subject). The image captured by the imaging device is input to the learning data generation device 100. At this time, the imaging device and the learning data generation device 100 may be connected by a wireless or wired communication network so that the captured image data can be directly imported from the imaging device, or the captured image data may be stored from the imaging device in an external storage device and the external storage device may be connected to the learning data generation device 100 so that the captured image data can be imported.

The training data generating device 100 includes a first control device 110, a second control device 120, a third control device 130, a segmentation image output unit 140, and a storage device 150.

The first control device 110 includes an image data receiving unit 111 that receives input of image data of a learning subject and stores the data in a memory unit 150, a color similarity calculation unit 112, and a pattern classification unit 113. The first control device 110 corresponds to the first function of the present embodiment described above.

Figure 3 is a diagram for explaining a method for calculating color similarity and pattern classification using color similarity. The color similarity calculation unit 112 calculates color similarity between image data based on the color feature distribution of the image data. The color feature distribution is the occurrence frequency distribution of hue, saturation (saturation/chroma), and value (value/brightness) in the HSV color space. Image data has different color features depending on the light environment, including natural light and/or artificial light, in the shooting environment. For example, the occurrence frequency of each of the hue, saturation, and value of each pixel is calculated to create a histogram, and the color similarity between image data can be calculated by comparing the histograms (distribution status).

The HSV color space is a color space consisting of three components: hue, saturation, and lightness. Hue is the type of color (e.g. red, blue, yellow) and is divided into hues from 0 to 360. Saturation is the vividness of a color, and as the saturation of a color decreases, it becomes more gray and dull. Lightness is the brightness of a color. Hue in the HSV color space is information with saturation (0 to 100) and lightness (0 to 100) as parameters, and one pixel has color information (pixel value) of, for example, hue "36", saturation "80", and lightness "100".

By calculating the frequency of occurrence of each of the hue, saturation, and brightness and generating a histogram, it is possible to grasp the color feature distribution, and then by using a known histogram similarity calculation method, it is possible to grasp the similarity of the color feature distribution of the image data.

The pattern classification unit 113 groups (patterns) similar images based on the calculated color similarity, and forms an image data group for each pattern. Image data groups with similar color similarity have a common color feature distribution, for example, due to similar lighting conditions in the shooting environment. The automatic classification by patterning in this embodiment makes it possible to form a collection of image data with similar color feature distributions.

Once each pattern classification is formed, it becomes an individual group with a certain degree of dissimilarity to each other, and image data input thereafter is sorted into groups with a certain range of similarity, or if there is no similarity within a certain range with any of the groups, it is sorted into a new pattern.

Next, the second control device 120 includes a color adjustment parameter setting unit 121 and a color adjustment unit 122. The color adjustment parameter setting unit 121 accepts input of color adjustment parameters that are set for each pattern classification, taking into account the shooting environment.

The color adjustment parameters are parameters that correct the chromaticity strength of the RGB color space based on wavelengths according to the lighting environment, including natural and/or artificial light, in the shooting environment, or the difference in the HSV color space. For example, when comparing a part exposed to sunlight and a part not exposed to sunlight, the RGB elements of G (green) and B (blue) are more prominent in the part exposed to sunlight. Therefore, by setting color adjustment parameters that emphasize R (red) and performing color adjustment of the entire image data, the effects of the lighting environment can be reduced.

The user can set any value for the color adjustment parameters by referencing each image in the image data group for each pattern classification. The color adjustment parameter setting unit 121 can display an image viewing screen, a color adjustment parameter input screen, etc. on a specified display and accept color adjustment parameters through the screen for each pattern classification.

The third control device 130 includes a grayscale conversion unit 131, a brightness threshold setting unit 132, and an area designation unit 133. The grayscale conversion unit 131 converts the color-tone corrected image data into grayscale. The brightness threshold setting unit 132 accepts input of a brightness threshold for each pattern classification. The area designation unit 133 partitions an area in the grayscale converted image data based on the brightness threshold and designates the area.

Figure 4 is a diagram for explaining area designation, in which the second control device 120 and the third control device 130 perform the second function described above.

As shown in Figure 4, image data can be converted to grayscale and the contour of the area range can be formed based on a brightness threshold. Then, for example, the area surrounded by the contour created by the brightness threshold and the other area can be expressed in different colors to generate learning data with specified areas (segmented). In the example of Figure 4, the dark parts are the extracted areas whose contours are created by the brightness threshold, and the light parts are the other areas.

For example, if the extracted region has a strong tendency toward white, a contour made up of pixels above the brightness threshold can be formed, and if it has a strong tendency toward black, a contour made up of pixels below the brightness threshold can be formed. In other words, the brightness threshold is used differently depending on the relationship between the color of the learning subject (monitoring subject) and its surroundings, and a contour above the threshold or a contour below the threshold is formed to specify the region.

Figure 5 is a diagram showing the process flow of the learning data generation process. When image data captured by a photographing device is input (S101), color similarity is calculated (S102), and the data is sorted into pattern groups having a certain range of similarity based on the color feature distribution (S103).

Color adjustment parameters can be set for each sorted pattern classification (S104), and color adjustment processing is performed collectively for each pattern classification using the set color adjustment parameters (S105). The color-adjusted image data is converted to grayscale (S106), and area designation processing is performed using the grayscale image. Specifically, the brightness threshold used in the area designation processing can be set collectively for each pattern classification (S107), and the same brightness threshold is applied to each image of the same pattern classification.

Based on the brightness threshold, the contour of the learning range in the image data is formed, and the area is divided into an area above the brightness threshold and an area below the brightness threshold, or into an area inside the contour and an area outside the contour, to perform area designation (area extraction) (S108). Since a different brightness threshold is set for each pattern classification, area designation can be performed for each pattern classification all at once.

The learning data that has been subjected to the segmentation process in this manner is stored in the storage unit 150. The stored learning data is output via the segmentation image output unit 140 to a device for creating teacher data, etc. (S109).

According to this embodiment, by classifying the similarities between images into patterns based on the color feature distribution of each captured image data, it is possible to automatically perform sorting that takes into account the shooting environment (light environment). Then, for each sorted pattern classification, color adjustment processing that takes into account the shooting environment is performed all at once, and grayscale conversion is performed to narrow down the color range that can be used as the learning target area based on the difference in brightness due to color.

For the narrowed-down color range, the contours of the area are highlighted based on a brightness threshold set for each pattern classification, allowing for area specification, eliminating the need for manual range specification and enabling highly accurate segmentation to be performed automatically.

In this way, it is possible to generate a large amount of highly accurate training data that has been subjected to segmentation processing in a short period of time.

Although the embodiment has been described above, the training data generation device 100 can be configured as part of a learning system equipped with a learning model such as AI, machine learning, or deep learning, or as part of a training data generation system for providing training data to a learning system. For example, the training data generation device 100 can become a training data generation system if it is configured to have a function of selecting image data that has been subjected to segmentation processing and assigning training labels.

The learning data generation device 100 is a computer device equipped with the computational functions, storage functions, and communication functions of a server device or the like. The hardware configuration may include a memory (main storage device), operation input means such as a mouse, keyboard, touch panel, and scanner, output means such as a printer, and an auxiliary storage device (hard disk, etc.).

Furthermore, each function of the present invention can be realized by a program, and a computer program prepared in advance to realize each function is stored in an auxiliary storage device, and a control unit such as a CPU reads the program stored in the auxiliary storage device into a main storage device, and the control unit executes the program read into the main storage device, thereby causing the computer to operate the functions of each unit of the present invention. On the other hand, each function of the learning data generation device 100 can be configured as a separate device, or a computer system can be configured by connecting multiple devices directly or via a network.

The above program can also be provided to a computer in a state in which it is recorded on a computer-readable recording medium. Examples of computer-readable recording media include optical disks such as CD-ROMs, phase-change optical disks such as DVD-ROMs, magneto-optical disks such as MO (Magnet Optical) and MD (Mini Disk), magnetic disks such as floppy (registered trademark) disks and removable hard disks, and memory cards such as Compact Flash (registered trademark), Smart Media, SD memory cards, and memory sticks. Also included as recording media are hardware devices such as integrated circuits (IC chips, etc.) that are specially designed and configured for the purposes of the present invention.

Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

100 Learning data generating device 110 First control device 111 Image data receiving unit 112 Color similarity calculation unit 113 Pattern classification unit 120 Second control device 121 Color adjustment parameter setting unit 122 Color adjustment unit 130 Third control device 131 Grayscale conversion unit 132 Brightness threshold setting unit 133 Area designation unit 140 Segmentation image output unit 150 Storage unit

Claims (5)

a color similarity calculation unit that calculates a color similarity between image data obtained by photographing a learning object based on a color feature distribution of the image data;
a pattern classification unit that classifies the image data into a plurality of patterns using the color similarity;
a color adjustment unit that corrects the color tone of the image data using color adjustment parameters that are set for each pattern classification in consideration of the shooting environment;
A conversion unit that converts the color-corrected image data into grayscale data;
a region designation unit that partitions regions in the grayscale converted image data based on a luminance threshold value that is set for each pattern classification, and designates the regions;
A learning data generation system comprising: The image data of the learning object has different color features depending on a light environment including natural light and/or artificial light in the shooting environment,
The color feature distribution is a distribution of occurrence frequencies of hue, saturation, and brightness,
The learning data generation system according to claim 1, characterized in that the color similarity calculation unit calculates the occurrence frequency distribution of each of hue, saturation, and lightness, and calculates the color similarity by comparing the occurrence frequency distributions between image data. the color adjustment parameter is a parameter for correcting the chromaticity intensity in the RGB color space or the difference in the HSV color space based on the wavelength according to the light environment including natural light and/or artificial light in the shooting environment,
The training data generation device according to claim 1 or 2, characterized in that the area designation unit forms a contour of a learning target range in the image data based on the brightness threshold, and designates the area by dividing the area into an area above the brightness threshold and an area below the brightness threshold, or into an area inside the contour and an area outside the contour. A program executed by a computer,
A first function of calculating a color similarity between image data obtained by photographing a learning object based on a color feature distribution of the image data;
A second function of classifying the image data into a plurality of patterns using the color similarity;
a third function of correcting the color tone of the image data using color adjustment parameters that are set for each pattern classification in consideration of the shooting environment;
A fourth function of converting the color-corrected image data into a grayscale image;
A fifth function of partitioning an area in the grayscale converted image data based on a luminance threshold value set for each pattern classification and specifying the area;
A program for causing the computer to realize the above. an image data receiving unit that receives input of image data obtained by photographing a learning subject;
a color similarity calculation unit that calculates a color similarity between image data based on a color feature distribution of the image data;
a pattern classification unit that classifies the image data into a plurality of patterns using the color similarity;
a color adjustment unit that receives input of color adjustment parameters that are set for each pattern classification in consideration of the shooting environment, and corrects the color tone of the image data using the set color adjustment parameters;
A conversion unit that converts the color-corrected image data into grayscale data;
an area designation unit that accepts an input of a luminance threshold value set for each pattern classification, and partitions an area in the grayscale converted image data based on the luminance threshold value to designate the area;
a learning data output unit that outputs image data with a specified region;
A learning data generation system comprising:

Images