JP2019215698A - Image inspection support apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image inspection support device capable of saving labor for confirming an error in a label attached to an image and reducing man-hours and labor required for the confirmation. An image inspection support apparatus 1 constructs a neural network by learning an image used for learning of an image inspection apparatus that determines the quality of a product, and uses a learned neural network to generate an image for determination. Extract feature quantities. The image inspection support device 1 calculates the degree of similarity, which is an index value indicating the correlation between the images, based on the feature amount of the extracted determination images. Then, the image inspection support apparatus 1 determines whether or not confirmation is required to correct the label attached to the target determination image based on the calculated similarity and the label attached to the determination image. To judge. [Selection diagram]

Description

  The present invention relates to an image inspection support apparatus and method, and more particularly, to a technique for supporting confirmation of a label attached to an image used for image inspection.

  2. Description of the Related Art Conventionally, as a technique for automating inspection of a product visually performed by a person, an image inspection for determining whether a product is a non-defective product or a non-defective product using an image to be inspected is known.

  In recent years, with the progress of machine learning represented by deep learning, a learning-type image inspection technique has been developed. In this method, the difference between the characteristics of a good product and a bad product is learned using an image in which the distinction between the good product and the bad product is labeled in advance, and a product inspection is performed using the learned result. This method has an advantage that an image inspection can be performed if an image for learning is prepared without deep knowledge about an inspection object and image processing.

  Learning requires images labeled with good or bad discrimination. In many cases, such data cannot be automatically generated, and it is often necessary to label images using the result of a visual inspection by a person.

  When machine learning is used for an algorithm in an image inspection system that determines whether a product is a good product or a defective product using an image, the accuracy of the label attached to the image greatly affects the determination performance. However, when labeling images is performed manually as described above, errors in labeling are likely to occur. Also, once an image has been labeled, it is unlikely that the directly labeled image will be confirmed in subsequent steps. For this reason, the user may use the information for learning without noticing the mistake, which may greatly affect the discrimination performance.

Pascal Vincent, Hugo Larochel, Isabelle Lajoie, Yoshia Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11 (Dec): 3371-3408, 2010. L. J. P. van der Maaten and G. E. Hinton. Visualizing High-Dimensional Data Usage t-SNE. Journal of Machine Learning Research 9 (Nov): 2579-2605, 2008.

  In a conventional learning-type image inspection system, in order to suppress a decrease in discrimination accuracy due to a labeling error, an operation of confirming such labeling error is required. Generally, machine learning using deep learning requires a large amount of data. Since a large amount of image data is prepared even in a learning type image inspection system, it is necessary to check all of these images one by one. However, there is a problem that a large number of man-hours and labor are required for an operator to visually check this large amount of images.

  SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and can reduce the labor required to confirm an error of a label attached to an image, and can reduce the man-hour and labor required for the confirmation. Provide a support device.

  In order to solve the above-described problem, an image inspection support device according to the present invention is an image inspection support device that supports confirmation of a label attached to an image used in learning of a learning-type image inspection device. And an image acquisition unit that acquires an image used for determination, and a neural network that performs learning of a neural network based on the acquired learning image, and outputs a feature amount that can restore the learning image. Using the learning unit to be constructed and the learned neural network, the feature amount of the acquired determination image is extracted, and the target determination image and other determination images calculated from the feature amount are extracted. Based on the index value indicating the mutual correlation with each of the images, the necessity of confirmation for correcting the label attached to the target image for determination is determined, and the determination is made. And a determination section for outputting an image for determination of the obtained is characterized in that it comprises a label that has previously been granted.

  Further, in the image inspection support apparatus according to the present invention, the learning unit receives the learning image as an input, calculates a convolutional layer, and extracts a feature amount of the learning image. And performing an operation of a deconvolution layer having an inverse structure corresponding to the convolution layer, using the feature amount extracted by the first feature amount extraction unit as an input, and obtaining a restored image in which the learning image is restored. An image restoration unit to be output, an error calculation unit that calculates an error between the learning image input to the first feature amount extraction unit, and the restored image, and the calculated error is reduced. An error back-propagation unit that learns the convolutional layer and the deconvolutional layer using an error back-propagation method, wherein the determination unit is obtained by learning by the learning unit using the image for determination as an input. Learned tatami mats A second feature value extraction unit that performs an operation of the embedded layer and extracts a feature value of the image for determination, and a similarity that calculates the index value based on the feature value extracted by the second feature value extraction unit. The degree calculation unit, from the index value, there is the other determination image similar to the target determination image, and a label attached to the similar other determination image. When the label attached to the image for determination as the target is different from the label, the analysis unit determines that the label attached to the image for determination as the target needs confirmation for correction. And a result output unit for outputting a determination result by the analysis unit.

  Further, in the image inspection support device according to the present invention, the analysis unit may include, when there is no other determination image similar to the target determination image, the target determination image. It may be determined that the attached label may be confirmed for correction.

  Further, in the image inspection support device according to the present invention, the analysis unit is a similarity storage unit that stores the index value calculated by the similarity calculation unit for each image for determination, A threshold value setting unit that sets a threshold value, the index value stored by the similarity storage unit, the threshold value, and a label attached to the image for determination, A labeling determination unit that determines whether confirmation is necessary to correct the label attached to the target determination image, and wherein the similarity storage unit includes, for each of the determination images, the index Using a value, a predetermined number is selected in order from the other determination image having a higher similarity to the target determination image, and the selected other determination image is stored together with a label. The labeling determination unit determines the target The index value calculated for the image is compared with the threshold value, and based on the result of the comparison, it is determined whether or not confirmation for correcting a label attached to the target determination image is necessary. It may be determined.

  Further, in the image inspection support device according to the present invention, the labeling determination unit includes a label attached to the image for determination as the target and the label for another determination stored in the similarity storage unit. When the label attached to the image is different from the label attached to the image, it may be determined that the label attached to the target determination image needs to be confirmed for correction.

  Further, in the image inspection support apparatus according to the present invention, the result output unit may include a feature amount compression unit that compresses a dimension of the feature amount extracted by the second feature amount extraction unit; A feature amount visualization unit that generates a scatter diagram for each image for determination and displays the scatter diagram on a display device, and an operation unit that externally receives a label correction instruction for the target image for determination displayed on the display device And a data correction unit that corrects a label attached to the target image for determination based on the correction instruction and stores the corrected data in a storage unit.

  Further, an image inspection support method according to the present invention is an image inspection support method for supporting confirmation of a label attached to an image used in learning of a learning-type image inspection apparatus, wherein an image used for learning and determination is determined. Acquiring an image acquiring step, learning a neural network based on the acquired learning image, and constructing a learned neural network that outputs a feature amount that can restore the learning image, Using the trained neural network, the feature amount of the acquired image for determination is extracted, and calculated from the extracted feature amount, each of the target determination image and the other determination image. Based on the index value indicating the correlation between the two, the necessity of confirmation for correcting the label attached to the target image for determination is determined, and a determination result is output. And a determination step of an image for determination of the obtained is characterized in that it comprises a label that has previously been granted.

  According to the present invention, an index value indicating the correlation between images is calculated from the feature amount of an image for determination extracted by a neural network learned based on an image, and based on the index value, an index value is calculated based on the index value. Since the necessity of confirmation for correcting the attached label is determined, the work of confirming the error of the label attached to the image can be saved, and the man-hour and labor required for the confirmation can be reduced.

FIG. 1 is a diagram illustrating the principle of the image inspection support device according to the present invention. FIG. 2 is a diagram illustrating the principle of the image inspection support device according to the present invention. FIG. 3 is a functional block diagram of the image inspection support device according to the first embodiment of the present invention. FIG. 4 is a block diagram illustrating a hardware configuration of the image inspection support device according to the first embodiment of the present invention. FIG. 5 is a flowchart illustrating the overall operation of the image inspection support device according to the first embodiment of the present invention. FIG. 6 is a flowchart illustrating a learning process according to the first embodiment of the present invention. FIG. 7 is a flowchart illustrating a determination process according to the first embodiment of the present invention. FIG. 8 is a functional block diagram of the image inspection support device according to the second embodiment of the present invention. FIG. 9 is a flowchart illustrating a determination process according to the second embodiment of the present invention. FIG. 10 is a functional block diagram of the image inspection support device according to the third embodiment of the present invention. FIG. 11 is a flowchart illustrating a determination process according to the third embodiment of the present invention. FIG. 12 is a flowchart illustrating a display process according to the third embodiment of the present invention. FIG. 13 is a diagram illustrating a display example by the result output unit according to the third embodiment of the present invention.

[Principle of the invention]
First, the principle of the present invention will be described.
In an image used for learning-type image inspection learning, there is often another image very similar to each image. If the similar images have the same labeling, it is very unlikely that they are incorrectly labeled. The reason is that, even if an operator might accidentally misidentify an image, it is unlikely that the same error is repeated many times for another similar image.

  Using this idea, if images similar to each other exist and the labels attached to those images are constant, it is not necessary to check the labels attached to all the images. . On the other hand, since only the labels of the images that do not satisfy this condition need to be checked, the work of checking the labeling error by the operator is greatly reduced.

  FIG. 1 is an explanatory diagram summarizing the principle of the present invention. If there are several similar images and the labels are consistent between the images, it is not necessary to check the labels attached to all similar images. If there are similar images and the labels are not constant among the images, there is a possibility that different labels are assigned between the similar images, and it is necessary to confirm the labels of those images. Also, when the number of similar images is very small, it is not possible to refer to the labels attached to the similar images for the necessity of confirming the labels attached to the target image. We cannot guarantee. Also in this case, it is preferable to check the label of the target image.

  By the way, in order to realize the above-described method of confirming an error in a label attached to an image, it is necessary to automatically find an image that is visually similar to a human. Therefore, the inventors have paid attention to a neural network as shown in FIG. 2 that can learn only with images without depending on label information.

  Examples of the neural network shown in FIG. 2 include an auto-encoder that learns to restore an input image, particularly a convolutional self-encoder and a hostile generation network (GAN). These neural networks learn the neural network so that the restored image is as close as possible to the input image. Therefore, it can be expected that the variables generated by these neural networks are feature amounts that reflect whether or not they are similar to each other visually. By determining whether or not the image is similar based on the feature amount, it is possible to extract an image that a person visually feels similar.

  Here, it is significant that the feature amount is generated by the neural network learned only with the image. Similar features exist in the learning-type image inspection system, but since these features are generated by learning the labeled data, they are features dependent on the label. Not suitable for use in finding mistakes.

  In addition, if a person manually sets a feature value based on experience or the like, it may take time to develop and adjust the feature value, which is not compatible with the object of the present invention of reducing man-hours and labor. Since the present invention uses a feature generated by a neural network learned only with images, it does not depend on labels and does not require knowledge of products. Therefore, it is suitable for automatically detecting an image with incorrect labeling.

The features of the present invention described above can be summarized in the following two points.
(I) For each learning image, the presence or absence of a similar image and the consistency of the label attached to the similar image are checked, and images that need to be confirmed are narrowed down.
(Ii) Whether or not the image is a similar image is determined based on a feature amount extracted by a neural network that can be learned only by an image without depending on a label.

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to FIGS.

[First Embodiment]
FIG. 3 is a functional block diagram of the image inspection support device 1 according to the first embodiment of the present invention.
The image inspection support device 1 learns an image used for learning of the image inspection device that determines the quality of a product, constructs a neural network, and extracts a feature amount of the image for determination using the learned neural network. I do. The image inspection support apparatus 1 calculates a similarity, which is an index value indicating the correlation between the images, based on the extracted feature amount of the image for determination. Then, based on the calculated similarity and the label attached to the image for determination, the image inspection support apparatus 1 determines whether or not confirmation for correcting the label attached to the target image for determination is necessary. Is determined.

[Functional blocks of the image inspection support device]
The image inspection support device 1 includes an image acquisition unit 10, a preprocessing unit 11, a storage unit 12, a learning unit 13, and a determination unit 14.

  The image acquisition unit 10 acquires an image used for learning by a learning-type image inspection device (not shown) that determines the quality of a product from the image inspection device or an external storage device via a communication network. More specifically, the image acquisition unit 10 acquires a learning image used by the learning unit 13 described later for learning. In addition, the image acquisition unit 10 acquires an image for determination, which is a target for which a determination unit 14 described later determines whether or not label confirmation is necessary. A label is given to the image for determination in advance.

  The learning images acquired by the image acquiring unit 10 generally include all images to be checked for labeling. Note that the image acquisition unit 10 may acquire, as learning images, different images in which the same inspection target is imaged. For example, it is also possible to increase the amount of learning data in the image inspection support apparatus 1 by adding an unlabeled image to the same inspection target.

  The preprocessing unit 11 performs preprocessing of the image acquired by the image acquisition unit 10 as necessary. More specifically, the pre-processing unit 11 removes noise by filtering or the like and normalizes the brightness of the image acquired by the image acquisition unit 10.

  The storage unit 12 stores the image acquired by the image acquisition unit 10 and subjected to the preprocessing by the preprocessing unit 11. Further, the storage unit 12 stores information on a neural network used by the learning unit 13 and the determination unit 14 described later, and calculation results such as extracted feature amounts. In addition, the storage unit 12 stores preset determination criteria used by the analysis unit 142 described later.

  The learning unit 13 includes a first feature amount extraction unit 130, an image restoration unit 131, an error calculation unit 132, and an error back propagation unit 133.

  The learning unit 13 learns a feature amount of a learning image. More specifically, the learning unit 13 is an auto-encoder (self-encoder) that is a neural network capable of learning without supervision without depending on labels, and among them, a convolution-type self-encoder having a convolution layer (Non-patent Document 1) ) Is used to learn the feature amount of the image.

  The self-encoder used by the learning unit 13 is a neural network that learns to restore input image data. The self-encoder has the feature that no label is required for learning, and the present invention focuses on this point.

  The first feature extraction unit 130 is configured by at least one convolutional layer, and the image restoration unit 131 is configured by at least one deconvolutional layer, and implements a convolutional self-encoder of the learning unit 13. The deconvolution layer forming the image restoration unit 131 has an inverse structure corresponding to the convolution layer forming the first feature amount extraction unit 130.

  The first feature amount extraction unit 130 receives the learning image acquired by the image acquisition unit 10 as an input, performs a convolution layer operation, and extracts the feature amount of the input image. The feature amount extracted by the first feature amount extraction unit 130 is used later by the determination unit 14 to determine whether or not labeling confirmation is necessary.

  The image restoration unit 131 receives the feature amount extracted by the first feature amount extraction unit 130, performs an operation on the deconvolution layer, and outputs a restored image obtained by restoring a learning image.

The error calculation unit 132 and the error backpropagation unit 133 make the first feature amount extraction unit 130 and the image restoration unit 131 learn.
The error calculating unit 132 calculates an error between the image input to the first feature amount extracting unit 130 and the restored image restored by the image restoring unit 131.

  The error back propagation unit 133 performs learning of the first feature amount extraction unit 130 and the image restoration unit 131. More specifically, the error back-propagation unit 133 performs the convolutional self-encoding of the first feature amount extraction unit 130 and the image restoration unit 131 such that the error of the image calculated by the error calculation unit 132 is sufficiently small. Adjust the value of the vessel parameters. The error back-propagation unit 133 performs learning using, for example, a well-known error back-propagation method.

  The error calculation unit 132 and the error back-propagation unit 133 repeatedly perform learning until the error between the image input to the first feature amount extraction unit 130 and the image restored by the image restoration unit 131 becomes sufficiently small. If the error becomes sufficiently small, the learning is completed.

The determination unit 14 includes a second feature amount extraction unit 140, a similarity calculation unit 141, an analysis unit 142, and a result output unit 143.
The determination unit 14 determines whether or not confirmation for correcting a label attached to the image for determination is necessary using the feature amount of the image learned by the learning unit 13.

  The second feature amount extraction unit 140 receives as input an image to be determined whether or not to confirm the label acquired by the image acquisition unit 10 and extracts the feature amount of the image. More specifically, the second feature amount extraction unit 140 performs an operation on the convolutional layer learned by the learning unit 13 and extracts the feature amount of the image for determination. That is, the second feature amount extraction unit 140 uses the learned first feature amount extraction unit.

  The similarity calculating unit 141 is a similarity that is an index value indicating the correlation between the target determination image and each of the other determination images based on the feature amount of the image extracted by the second feature amount extraction unit 140. Is calculated.

  More specifically, when the dimension of the feature extracted by the second feature extracting unit 140 is too large, the similarity calculating unit 141 may calculate the similarity after compressing the dimension of the feature. Good. Thus, when the dimension of the feature amount extracted by the second feature amount extraction unit 140 is too large, it is possible to prevent the subsequent processing from requiring an enormous amount of calculation time. Specifically, the similarity calculation unit 141 performs dimension compression of the feature amount by Principal Component Analysis (PCA). If unnecessary, this process may be omitted.

  The similarity calculated by the similarity calculator 141 is an index used when determining whether two images are similar. In the present embodiment, a description will be given assuming that two images are more similar to each other as the degree of similarity between the images is higher.

  As the similarity between the images calculated by the similarity calculation unit 141, for example, one that can quantify the correlation between two data, such as a Euclidean distance and a Mahalanobis distance, is used. The similarity calculation unit 141 may calculate the similarity between images in advance for all of the images acquired by the image acquisition unit 10 and store the similarity in the storage unit 12. Further, the similarity may be calculated each time the value of the similarity is required in the analysis unit 142 described later. Note that the similarity calculation unit 141 may calculate an index value corresponding to the above-described similarity.

  The analysis unit 142 determines the label attached to the target determination image based on the similarity between the determination images calculated by the similarity calculation unit 141 and the label attached to the determination image. It is determined whether or not confirmation for correcting is necessary.

  The analysis unit 142 does not need to confirm the label attached to the image based on the predetermined condition regarding the calculated similarity, needs to confirm the label attached to the image, or performs the image It is determined that the label attached to may be checked.

  The analysis unit 142 satisfies a predetermined reference value among the determination images other than the target determination image acquired by the image acquisition unit 10 with respect to the target determination image. Whether or not there is an image having the similarity is used as the predetermined condition.

  More specifically, if there are other determination images having similarities that satisfy a preset reference value, and if all the other determination images have the same label, The analysis unit 142 determines that it is unnecessary to confirm the label in the target determination image because the label attached to the target determination image is unlikely to be incorrect. That is, when the same label is assigned to images similar to each other, it is not necessary to reconfirm the label of the target determination image.

  In addition, there are other determination images having a similarity satisfying a preset reference value, and among those other determination images, the label of the target determination image is If there is an image with a different label, the analysis unit 142 determines that it is necessary to check the label of the image because the label attached to the target determination image is likely to be incorrect. judge. That is, when different labels are attached to images similar to each other, it is necessary to reconfirm the label of the target determination image.

  If there is another image for determination having a similarity that satisfies a preset reference value and the number of other images for determination is less than or equal to a predetermined number, the analysis unit 142 Since there are not enough images similar to the determination image, it is analyzed that it is not possible to determine whether or not to confirm the labeling. In this case, the analysis unit 142 does not strongly recommend reconfirmation of the label attached to the target determination image, but cannot reject the possibility that a potential labeling error is included. It is determined that the label may be confirmed in accordance with. It should be noted that even when there is no other image for determination having the similarity satisfying the reference value, the image is similarly determined.

  The result output unit 143 outputs the determination result by the analysis unit 142. More specifically, the result output unit 143 displays, on a predetermined screen of the display device 107, which will be described later, at least information on the image for which the analysis unit 142 has determined that the label needs to be confirmed. The result output unit 143 may display on a predetermined screen of the display device 107 information on the image determined by the analysis unit 142 to be able to confirm the label as necessary. The result output unit 143 may display the determination result on the screen of the display device 107, or may record it as a file or a database in a predetermined area of the storage unit 12.

[Hardware configuration of image inspection support device]
Next, a hardware configuration of the image inspection support device 1 having the above-described functions will be described with reference to FIG.
The image inspection support apparatus 1 includes a computer including a computing device 102 having a CPU 103 and a main storage device 104 connected via a bus 101, a communication control device 105, an external storage device 106, a display device 107, an input device 108, and the like. , Can be realized by a program that controls these hardware resources.

  The CPU 103 and the main storage device 104 constitute the arithmetic unit 102. A program for the CPU 103 to perform various controls and calculations is stored in the main storage device 104 in advance. Note that the arithmetic device 102 may use a GPU (Graphics Processing Unit) as necessary, in addition to the CPU 103.

  The communication control device 105 is a control device for connecting the image inspection support device 1 and various external electronic devices via a communication network NW. The communication control device 105 may receive a labeled image from an image inspection device, a server, or the like installed outside via the communication network NW. In addition, the communication control device 105 transmits a determination result indicating whether or not it is necessary to confirm the label attached to the determination image obtained by the image inspection support device 1 to an external device or the like via the communication network NW. May be.

  The external storage device 106 includes a readable and writable storage medium, and a drive device for reading and writing various information such as programs and data on the storage medium. For the external storage device 106, a semiconductor memory such as a hard disk or a flash memory can be used as a storage medium. The external storage device 106 is an image data storage unit 106a, a setting information storage unit 106b, a learning information storage unit 106c, a determination information storage unit 106d, a program storage unit 106e, and other storage devices (not shown). Storage device for backing up programs, data, and the like stored therein.

  The image data storage unit 106a stores an image acquired by the image acquisition unit 10 described with reference to FIG. 3 from an external image inspection device, a storage device, or the like. The image data storage unit 106a stores image data including an image, a label assigned to the image, and a title of the image.

  The setting information storage unit 106b stores various kinds of information on the neural network used by the learning unit 13 and the determination unit 14 described with reference to FIG. The setting information is stored.

The learning information storage unit 106c stores information such as feature amounts extracted by the learning unit 13 described with reference to FIG.
The determination information storage unit 106d stores the determination result by the determination unit 14 described with reference to FIG.
Note that the image data storage unit 106a, the setting information storage unit 106b, the learning information storage unit 106c, and the determination information storage unit 106d correspond to the storage unit 12 described with reference to FIG.

  The program storage unit 106e stores various programs for the image inspection support apparatus 1 to execute arithmetic processing necessary for image inspection support, such as image acquisition processing, learning processing, similarity calculation processing, and determination processing in the present embodiment. Is stored.

  The display device 107 constitutes, for example, a display screen of the result output unit 143 described with reference to FIG. The display device 107 is realized by a liquid crystal display or the like.

  The input device 108 includes a keyboard, a mouse, a touch panel, and the like. The input device 108 receives a key input by an operator, a click of an icon or the like, a function or an operation input corresponding to a touch position on the touch panel.

[Operation of image inspection support device]
The operation of the image inspection support device 1 having the above-described configuration will be described with reference to the flowcharts of FIGS.

First, the entire operation of the image inspection support device 1 will be described with reference to FIG.
The image acquisition unit 10 acquires a plurality of images to be inspected from an external image inspection device, a storage device, or the like (Step S1). Next, the preprocessing unit 11 performs filtering, normalization, and the like of the image acquired by the image acquisition unit 10, and performs preprocessing of the image as needed (step S2). The preprocessed image is stored in the storage unit 12.

  Note that the learning images acquired by the image acquisition unit 10 include all determination images to be checked for labeling. Hereinafter, label information given in advance is included for learning and for determination, and the image for learning and the image for determination may be simply referred to as “image”.

  Next, when the learning unit 13 has already learned the feature amount extractor for extracting the feature amount of the image (step S3: YES), the arithmetic unit 102 of the image inspection support device 1 makes the determination by the determination unit 14. The process proceeds to Step S5. On the other hand, if the feature amount extractor has not been learned yet (step S3: NO), the process proceeds to the learning process by the learning unit 13 (step S4).

  When the learning process is performed by the learning unit 13 (step S4), a determination process is performed by the determination unit 14 to determine whether confirmation for correcting the label attached to the image for determination is necessary. (Step S5), the process ends.

[Learning process]
Next, the learning process (step S4) by the learning unit 13 will be described in more detail with reference to FIG.
First, the first feature value extraction unit 130 extracts a feature value of an image using the learning image acquired by the image acquisition unit 10 as an input (step S40). Next, the image restoration unit 131 restores the image using the feature amount of the image extracted in step S40 as an input (step S41).

  After that, the error calculation unit 132 calculates an error between the image input to the first feature amount extraction unit 130 and the restored image output from the image restoration unit 131 (Step S42). Next, the error back-propagation unit 133 uses a well-known error back-propagation method to reduce the error between the input image calculated by the error calculation unit 132 and the restored image. Learning of the convolutional layer and the deconvolutional layer constituting the image restoration unit 130 and the image restoration unit 131 is performed (step S44).

  When the convolution layer and the deconvolution layer are learned by the error back propagation unit 133 and the parameter values are updated, the processing from step S40 to step S44 is repeated and executed again. After that, when the error between the input image and the restored image, which has been calculated again, is sufficiently small (step S45: YES), the learning is completed, and the error calculating unit 132 proceeds to the determining process by the determining unit 14 (step S5). I do.

  The values of the parameters of the neural network including the convolution layer and the deconvolution layer that are updated and finally determined by the error back propagation unit 133 so as to reduce the error between the input image and the restored image are information indicating the feature amount. Is stored in the storage unit 12.

[Determination process]
Next, the determination process (step S5) executed by the determination unit 14 will be described in more detail with reference to FIG.
First, the second feature amount extraction unit 140 receives the image for determination acquired by the image acquisition unit 10 in step S1, performs an operation on the convolutional layer obtained by learning by the learning unit 13, and performs processing on the image for determination. A feature amount is extracted (step S50).

  Next, before calculating the similarity, the similarity calculation unit 141 performs dimensional compression on the feature amount of the image for determination extracted in step S50 as needed (step S51). The similarity calculation unit 141 may perform dimension reduction of the feature amount by PCA.

  Next, the similarity calculation unit 141 calculates the similarity between the images for determination based on the feature amount whose dimensions have been compressed in step S51 (step S52). The similarity calculating unit 141 calculates, for each image for determination, a similarity, which is an index indicating a correlation between the images for determination, using the Euclidean distance, for example. More specifically, the similarity calculation unit 141 calculates the similarity of the feature amount of the image for determination that is extracted in step S50 and dimensionally compressed in step S51.

  Next, the analysis unit 142 determines whether it is necessary to confirm the label attached to the target determination image based on the similarity of the determination image calculated in step S52 (step S53). . More specifically, the analysis unit 142 does not need to confirm the label attached to the target determination image based on the predetermined condition regarding the similarity of the determination image calculated in step S52. It is determined that the label attached to the image needs to be confirmed, or the label attached to the image may be confirmed as necessary.

  Specifically, the analysis unit 142 reads out a predetermined condition used for the determination from the storage unit 12. The predetermined condition is that there is another determination image having a similarity satisfying a preset reference value with respect to the similarity calculated for the target determination image, or a certain number of images or less. Only exists.

  Then, another determination image having a similarity that satisfies a preset reference value with respect to the similarity of the target determination image calculated in step S52 exists in the storage unit 12, and If all the other determination images have the same label, the analysis unit 142 analyzes that the label attached to the target determination image is unlikely to be incorrect. In this case, the analysis unit 142 determines that it is not necessary to check the label of the target determination image.

  Further, another determination image having a similarity that satisfies a preset reference value with respect to the similarity of the target determination image calculated in step S52 exists in the storage unit 12, and In the case where an image with a different label is present in these other determination images, the analysis unit 142 determines that the label attached to the target determination image is likely to be incorrect. analyse. In this case, the analysis unit 142 determines that it is necessary to confirm the label of the target determination image.

  In addition, another image for determination having a similarity that satisfies a preset reference value with respect to the similarity of the target image for determination calculated in step S52 exists in the storage unit 12, If the number of other determination images is less than or equal to the predetermined number, the analysis unit 142 determines whether or not it is necessary to confirm labeling because there are not enough images similar to the target determination image. Analyze that it is not possible to judge. In this case, the analysis unit 142 determines that the labeling may be confirmed as necessary.

The determination result by the analysis unit 142 is stored in the storage unit 12.
Thereafter, the result output unit 143 outputs the determination result by the analysis unit 142 (Step S54). The result output unit 143 may, for example, display, on a predetermined screen of the display device 107, information on an image for which it has been determined that label confirmation is necessary.

  As described above, according to the first embodiment, the image inspection support device 1 learns the feature amount of the labeled image based on only the image, and uses the neural network constructed by the learning. The feature amount of the image for determination is extracted. Further, the image inspection support device 1 calculates the similarity between the images based on the extracted feature amount of the determination image, and determines the target determination image based on the similarity of the determination image. It is determined whether or not it is necessary to check the label attached to, and the result of the determination is output.

  Therefore, the work of checking the label of an image with incorrect labeling can be saved, and the number of steps and labor required for the check can be reduced. Further, by using the labeled image as the input image, the image inspection support apparatus 1 can automate the output of the determination result indicating whether the confirmation of the labeling is necessary. Therefore, the labor required for the operator to confirm the labeling can be significantly reduced.

  Further, the image inspection support apparatus 1 according to the present embodiment calculates the similarity between images using an algorithm including the Euclidean distance. The calculation of the similarity is simple as an algorithm, and the mounting of the determination unit 14 is facilitated.

  Further, the image inspection support apparatus 1 according to the present embodiment determines whether it is necessary to confirm the label attached to the target image for determination based on the similarity between the images, It is possible to present information including an error in attachment.

  Further, the image inspection support apparatus 1 according to the present embodiment needs to perform learning of the input image at least once by the learning unit 13, but once the feature amount extractor is learned from the learning image. Only the feature amount extraction of the image for determination by the learned feature amount extractor and the calculation of the similarity need be performed. Therefore, a check operation that is faster than a label check operation that is performed by all the operators visually is realized.

  In the embodiment described above, the analysis unit 142 determines that it is unnecessary to check the label attached to the target determination image, and checks the label attached to the target determination image. Has been described above, or the determination that the label may be confirmed as necessary has been described. However, the analysis unit 142 may adopt a configuration that detects only when it is necessary to confirm the label attached to the target determination image.

  In this case, the analysis unit 142 reads, for example, only images labeled as “defective” from the images acquired by the image acquisition unit 10 from the storage unit 12, and reads the images among images similar to each other. What is necessary is just to detect the presence of another image labeled as "good" and determine that the image needs to be checked for the label. In this configuration, the analysis unit 142 does not need to perform analysis or determination on all of the images acquired by the image acquisition unit 10, and has an advantage that the amount of calculation is smaller when the proportion of defective products is low. is there.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the following description, the same components as those in the above-described first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

  In the first embodiment, a case will be described in which the analysis unit 142 determines whether it is necessary to confirm a label attached to a target determination image based on a predetermined condition stored in the storage unit 12. did. On the other hand, in the second embodiment, the analysis unit 142A determines whether it is necessary to check the label attached to the target determination image by the threshold processing. In the image inspection support device 1A according to the second embodiment, the configuration other than the analysis unit 142A is the same as the configuration of the image inspection support device 1 according to the first embodiment. Hereinafter, a description will be given mainly of a configuration different from the first embodiment.

[Functional block of analysis unit]
The analysis unit 142A includes a similarity storage unit 420, a threshold setting unit 421, and a labeling determination unit 422.

  The similarity storage unit 420 selects a predetermined arbitrary number of images in the order of similarity between images based on the feature amount of each image calculated by the similarity calculation unit 141, in descending order of similarity, The label of the selected image and the similarity are stored in association with each other. Although the number of images selected by the similarity storage unit 420 can be selected as small as one and as many as the number of samples -1 at maximum, it can be practically used by selecting about 3 to 5 images.

  The threshold value setting unit 421 sets a threshold value of a feature amount that the two images are similar to each other. The threshold setting unit 421 may set a value that determines that the similarity of an image is lower based on the similarity distribution stored by the similarity storage unit 420 and the statistics thereof, as the threshold. Good.

  The threshold value setting unit 421 may set a threshold value suitable for the design. For example, the threshold value setting unit 421 sorts images having high similarities and determines the similarity having a rank corresponding to 1% of all image data. The threshold may be set. In addition, the threshold value setting unit 421 inputs a threshold value set by the operator based on a distribution graph or image display generated based on the similarity information stored in the similarity storage unit 420. It may be received via the device 108.

  The labeling determination unit 422 determines the similarity stored in the similarity storage unit 420, the threshold set by the threshold setting unit 421, and the label attached to the image for determination. It is determined whether it is necessary to confirm the label attached to the target determination image.

  More specifically, the labeling determination unit 422 determines that the label of the image having a high similarity to the target determination image is different from the label attached to the target determination image. It is analyzed that the labeling error of the target determination image is suspected, and it is determined that the label of the target determination image needs to be confirmed.

  If the similarity calculated by the similarity calculation unit 141 is lower than the threshold set by the threshold setting unit 421, the labeling determination unit 422 determines which other Since the similarity is low with the image for determination of, it is analyzed that there is a potential error in labeling in the target image for determination. In this case, the labeling determination unit 422 determines that the label may be checked as needed for the target determination image. Note that the labeling determination unit 422 may use a configuration in which an appropriate method is added according to the design and unnecessary methods are deleted.

  Further, the labeling determination unit 422 may select and output an image for which label confirmation should be preferentially selected from among images determined to require label confirmation. More specifically, if the labels of at least two or more determination images having a feature amount that is considered to have a high similarity to the feature amount of the target determination image, the label of the target determination image is used. When the label is different from the label of the image, there is a high possibility that the label of the target image for determination is wrong.

  Because, if the probability of making a mistake in individual labeling is the same, there is a higher possibility that only one image for determination as a target is wrong than that in which all labels of a plurality of other images are wrong. It is considered that. Therefore, by selecting and preferentially outputting an image that satisfies the above conditions, it is possible to check and correct the label more efficiently.

  In addition, the labeling determination unit 422 performs the determination again after the label is corrected for the image that is preferentially output as the image that needs to be checked for a label. If the label is determined again, the incorrect label is corrected. More efficient label confirmation and correction will be implemented.

[Determination process]
Next, a determination process performed by the determination unit 14A will be described with reference to a flowchart of FIG. Note that the overall flow and the learning process of the image inspection support device 1A are the same as in the first embodiment. Therefore, the determination process performed by the determination unit 14A will be described in detail below.

  First, the second feature amount extraction unit 140 extracts the feature amount of the image for determination using the learned neural network, that is, the feature amount of the image learned by the learning unit 13 using the learning image as an input ( Step S50A).

  Next, before calculating the similarity, the similarity calculating unit 141 performs dimensional compression on the feature amount of the image for determination extracted in step S50A as needed (step S51A). The similarity calculation unit 141 may perform dimension reduction of the feature amount by PCA.

  Next, the similarity calculation unit 141 calculates the similarity between the images based on the feature amount of the image to be determined whose dimensions have been compressed in step S51A (step S52A). Next, the similarity storage unit 420 selects, from among the similarities between the images calculated in step S52A, a feature amount corresponding to a predetermined arbitrary number of images in order from the image having the highest similarity, The label attached to the image, which is the original data of the feature, and the similarity are stored in association with each other (step S53A). More specifically, the similarity storage unit 420 stores the identification information of the image, the label attached to the image, and the similarity in association with each other.

  Next, the threshold value setting unit 421 sets a threshold value for the similarity of the image for determination stored by the similarity storage unit 420 (Step S54A). Thereafter, the labeling determination unit 422 performs threshold processing on the similarity of each of the determination images stored by the similarity storage unit 420, and checks the label attached to the target determination image. The necessity is determined (step S55A).

  More specifically, the labeling determination unit 422 detects a labeling error including a potential labeling error, and sets the similarity value calculated for the target determination image to a threshold value. If it is smaller, it is determined that it is necessary to check the label attached to the target determination image. Thereby, potential labeling errors can be detected.

  Further, when the value of the similarity calculated for the target determination image is equal to or greater than the threshold, the label of the target determination image and the label attached to another determination image Are matched. In this case, the labeling determination unit 422 determines that it is not necessary to check the label attached to the target determination image.

  On the other hand, when the value of the similarity is equal to or larger than the threshold, the labeling determination unit 422 determines that the label of the target determination image is different from the label of another determination image. Determines that it is necessary to confirm the label attached to the target determination image.

  As a result, it is possible to detect labeling errors other than potential errors where the value of the similarity is equal to or greater than the threshold value. The determination result by the labeling determination unit 422 is stored in the storage unit 12.

  After that, the result output unit 143 outputs the result of the determination by the labeling determination unit 422 (Step S56A).

  As described above, according to the second embodiment, the image inspection support device 1A calculates the similarity between the images for determination, and sets a threshold value for the calculated similarity. Then, based on the similarity in the feature amount of the determination image, the threshold value, and the label attached to the determination image, whether or not it is necessary to confirm the label attached to the target determination image is determined. Is determined. Therefore, the image inspection support apparatus 1A can save labor of checking an image with incorrect labeling, and can reduce man-hours and labor required for the check.

  Note that, in the embodiment described above, the case where the labeling determination unit 422 selects an image for which label confirmation is to be preferentially selected from among images for which labeling confirmation is necessary is described. did. In this case, for example, based on a label correction instruction received from a worker via the input device 108, the arithmetic unit 102 of the image inspection support device 1 corrects the label of the target image for determination, and Update the image data stored in. Thereafter, the determination process shown in FIG. 9 may be repeatedly executed. This realizes more efficient label confirmation and correction.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the following description, the same components as those in the above-described first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

  In the first and second embodiments, the determination results by the determination units 14 and 14A indicating the necessity of confirming the label attached to the target determination image are output from the result output unit 143. explained. On the other hand, in the third embodiment, a result output unit 143B including a user interface (hereinafter, referred to as “UI”) is provided, and the result output unit 143B displays the determination result, and issues a correction instruction by the operator. Accept.

  The configuration other than the result output unit 143B of the image inspection support device 1B according to the third embodiment is the same as that of the image inspection support devices 1 and 1A according to the first and second embodiments. In the following, a description will be given focusing on a configuration different from the first and second embodiments.

[Function block of result output unit]
As shown in FIG. 10, the result output unit 143B includes a feature amount compression unit 430, a feature amount visualization unit 431, a result detail display unit 432, a selection image display unit 433, an operation unit 434, and a data correction unit 435.

  The feature amount compression unit 430 compresses the feature amount of the image for determination extracted by the second feature amount extraction unit 140 stored in the storage unit 12 into one-dimensional to three-dimensional data. Usually, the feature amount of an image is high-dimensional, and it is difficult to display the similarity relationship of the feature amount on the screen of the display device 107 to the extent that a person can be identified. Therefore, the feature amount compression unit 430 compresses the feature amount into one-dimensional to three-dimensional data that can be displayed on a screen and can be identified by a person. If the dimension of the feature extracted by the second feature extractor 140 is too large, it may take an enormous amount of time to compress the feature. Therefore, dimension compression by PCA may be performed before the feature is compressed. .

  There are several methods for compressing the feature amount by the feature amount compression unit 430. More preferably, the feature amount is compressed using manifold learning (t-Distributed Stochastic Neighbor Embedding: t-SNE) (see Non-Patent Document 2). . The feature amount of the image compressed by the feature amount compression unit 430 is stored in the storage unit 12.

  The feature amount visualization unit 431 generates a scatter diagram of the feature amount of the image compressed from one-dimensional to three-dimensional by the feature amount compression unit 430, and visualizes the feature amount. More specifically, in the scatter diagram generated based on the data of the compressed feature amount, the feature amount visualizing unit 431 can distinguish each of the plots by reflecting the analysis result by the analyzing unit 142A and by color, shape, or the like. Create data.

  In addition, the feature amount visualization unit 431 links each plot with the corresponding image in the scatter diagram of the feature amount of the generated image. The feature amount visualization unit 431 displays an image corresponding to the selected plot and an image having a high degree of similarity to the selected plot in a predetermined screen area of the display device 107 in accordance with a plot selection operation in the scatter diagram. Thereby, the work of checking the label by the operator is made more efficient.

  The result detail display unit 432 displays image data including the title and label of the image stored in the storage unit 12, the determination result by the analysis unit 142A, and the similarity of the image stored by the similarity storage unit 420 and the similarity. A list of labels attached to the images is created and displayed in a predetermined screen area of the display device 107.

  The result detail display unit 432 displays, for example, information quantified in a stepwise manner as to the necessity of confirming the labeling as the determination result by the analysis unit 142A. For example, if the label on the image is suspected of being in error, and the image is determined to be required to be checked, "2", there is a possibility of a potential labeling error and check as necessary. The image determined to be acceptable is displayed as “1”. In addition, the result detail display unit 432 displays “0” for an image for which labeling is determined to be correct and confirmation is unnecessary.

  The result detail display unit 432 causes the display device 107 to display information on all images in descending order of numerical values representing the analysis results as described above, so that the operator can easily confirm an image to be corrected.

  The selected image display unit 433 receives a selection operation of a plot included in the scatter diagram of the feature amount of the image generated by the feature amount visualization unit 431, and an image corresponding to the selected plot and an image having a high degree of similarity to the image. Is displayed in a predetermined screen area of the display device 107.

  The operation unit 434 includes an icon, a text box, and the like displayed on the screen of the display device 107. The operation unit 434 receives, via the input device 108, a change in the threshold value set by the threshold value setting unit 421, a change in the number of images stored by the similarity storage unit 420, and a correction of a label.

  The data correction unit 435 corrects or changes information based on the correction or change operation received by the operation unit 434, and updates the information in the storage unit 12 or the similarity storage unit 420. The data correction unit 435 is realized by a correction button, a change button, and an end button displayed on the screen of the display device 107.

  Specifically, when an operation signal for instructing correction or change of the threshold value or the label is input from the operation unit 434, the data correction unit 435 reads the threshold value or the label information stored in the storage unit 12. Update. When the number of data stored in the similarity storage unit 420 is changed, the data correction unit 435 updates information in the similarity storage unit 420.

[Determination process]
Next, the operation of the image inspection support device 1 having the above-described configuration will be described. Note that the entire flow and the learning process of the image inspection support device 1 are the same as those in the first and second embodiments, and a description thereof will be omitted. Hereinafter, the determination process by the determination unit 14B will be described in detail.

  FIG. 11 is a flowchart illustrating a determination process according to the present embodiment. FIG. 12 is a flowchart illustrating the display processing. FIG. 13 is a diagram illustrating a display example by the result output unit 143B.

  First, the second feature amount extraction unit 140 extracts the feature amount of the image for determination using the learned neural network obtained by the learning unit 13 (step S50B).

  Next, before calculating the similarity, the similarity calculation unit 141 performs dimensional compression on the feature amount of the image for determination extracted in step S50B as needed (step S51B). The similarity calculation unit 141 may perform dimension reduction of the feature amount by PCA.

  Next, the similarity calculation unit 141 calculates the similarity of the feature amounts of the image for determination whose dimensions have been compressed in step S51B (step S52B). Next, among the similarities calculated in step S52B, the similarity storage unit 420 selects a feature amount corresponding to a predetermined arbitrary number of images in order from the image having the highest similarity, and The label attached to the image for determination as the original data and the similarity are stored in association with each other (step S53B).

  Next, the threshold value setting unit 421 sets a threshold value for the similarity of the image for determination stored by the similarity storage unit 420 (Step S54B). Thereafter, the labeling determination unit 422 performs a threshold process on the image for determination based on the similarity of the image stored by the similarity storage unit 420, and attaches it to the image for determination to be targeted. It is determined whether it is necessary to check the label (step S55B).

  Next, the result output unit 143B displays information on the determination result by the labeling determination unit 422 on the screen of the display device 107 (Step S56B).

Here, the display processing by the result output unit 143B will be described in more detail with reference to FIG.
First, the feature amount compression unit 430 performs dimension compression of the feature amount of the image stored in the storage unit 12 by PCA as necessary (step S560).

  Next, the feature compression unit 430 compresses the feature of the image that has been dimensionally compressed in step S560 into one-dimensional to three-dimensional data (step S561). Specifically, the feature compression unit 430 compresses the feature using the t-SNE (see Non-Patent Document 2). Next, the feature amount visualization unit 431 generates a scatter diagram of the feature amount data compressed in step S561, and displays the scatter diagram in a predetermined screen area of the display device 107 (step S562).

  Specifically, the feature amount visualization unit 431 displays the generated scatter diagram in the screen area a1 of the display device 107, as shown in FIG. In addition, the feature amount visualization unit 431 displays each plot in the scatter diagram by reflecting the result of the determination by the analysis unit 142A.

  For example, the feature amount visualization unit 431 specifies a display color of “red” in a style sheet (CSS) for a plot indicating an image determined to require label confirmation by the analysis unit 142A. In addition, the feature amount visualization unit 431 may specify a display color of “orange” for a plot indicating an image in which there is a possibility of a potential labeling error and recommendation of a label as necessary is recommended. do. In addition, the feature amount visualization unit 431 specifies a display color of “green” for a plot indicating an image for which labeling is correct and determination is not necessary.

  Next, the result detail display unit 432 creates a list of information including the title, label, similarity of the image stored by the similarity storage unit 420, and the determination result by the labeling determination unit 422, and generates a predetermined screen. It is displayed in the area (step S563). Specifically, the result detail display unit 432 displays a list in the screen area a2 as shown in FIG.

  Next, when receiving the selection of a predetermined plot from the outside in the scatter diagram displayed in step S562, the selected image display unit 433 displays the selected image in a predetermined screen area of the display device 107 (step S562). S564).

  Specifically, as shown in FIG. 13, the selected image display unit 433 displays an image corresponding to the plot selected by the operator via the input device 108 in the upper part of the screen area a3. The selected image display unit 433 displays the title of the image and its thumbnail in the upper part of the screen area a3. In addition, the selected image display unit 433 displays a title and a thumbnail of an image having a high similarity to the selected image in the lower part of the screen area a3.

  Returning to the determination process of FIG. 11, the operation unit 434 receives an operation on information about the determination result of the target determination image (selected image) displayed on the display device 107 from the outside (step S57B). The operation unit 434 receives, for example, an operation instruction input by the operator from the input device 108.

  Next, the operation instruction received by the operation unit 434 is an instruction to change the threshold for the similarity of the image set by the threshold setting unit 421, or the number of data stored by the similarity storage 420. If the instruction is a change instruction (step S58B: YES), the data correction unit 435 changes the threshold value or the number of data according to the instruction (step S59B).

  More specifically, the data correction unit 435 changes and stores the threshold value for the similarity of the image stored in the storage unit 12. Further, the data correction unit 435 sends an instruction to the similarity storage unit 420 to set the number of images stored by the similarity storage unit 420 to the number according to the change instruction. Steps S53B to S57B are repeatedly executed based on the data correction or correction instruction by the data correction unit 435.

  On the other hand, when the operation instruction received by the operation unit 434 is an operation instruction to correct the label attached to the selected image (Step S58B: NO, Step S60B: YES), the data correction unit 435 selects the selected image. The label of the image is corrected and stored in the storage unit 12 (Step S61B). After the data correction unit 435 corrects the label, a display process is performed by the result output unit 143B.

Here, with reference to a display example on the display device 107 shown in FIG. 13, an example of use of the image inspection support device 1B for the processing from step S57B to step S61B will be described.
As shown in FIG. 13, the operation unit 434 includes a threshold value change input unit I1, a storage number change input unit I2, and a label correction input unit I3. The operation unit 434 is realized by a text box provided in the screen area a4.

  The data correction unit 435 includes a change button I4, a correction button I5, and an end button I6. The data correction unit 435 is realized by a button provided in the screen area a4.

  The operator checks on the screen the selected image displayed on the screen area a3 of the display device 107, the image having a high degree of similarity to the selected image, and the analysis result displayed on the screen area a2. Then, the worker changes the threshold value for the similarity of the images or the number of images stored in the similarity storage unit 420 as necessary. In this case, the operator inputs a threshold value to be changed and a new image to the threshold value change input unit I1 or the storage number change input unit I2 displayed on the screen area a4 via the input device 108. Enter a number.

  Thereafter, the operator can confirm the change by clicking the change button I4 displayed on the screen area a4 via the input device 108.

  When correcting the label of the selected image displayed on the screen area a3, the operator inputs the label information to be corrected to the label correction input unit I3 displayed on the screen area a4 via the input device 108. input. Thereafter, the operator can fix the label correction by clicking the correction button I5 via the input device 108.

  After the label modification is confirmed, the end button I6 is clicked, thereby completing the label modification operation.

  As described above, according to the image inspection support device 1B according to the third embodiment, since the result output unit 143B is provided, the operator checks the determination result indicating whether the label check is necessary through the UI. , The label can be modified. By using such a UI, there is no need to search for image data to be corrected from a directory or a database containing a large amount of image data and perform label correction work, thereby facilitating label confirmation and correction work by an operator. It becomes.

  Further, since the image inspection support device 1B includes the operation unit 434 and can change the conditions used to determine whether or not it is necessary to confirm the label attached to the image as necessary, it is possible to easily perform trial and error. As a result, it is possible to save labor for confirming an image with incorrect labeling, and to reduce the man-hour and labor required for the confirmation.

  As described above, the embodiments of the image inspection support apparatus and the image inspection support method of the present invention have been described. However, the present invention is not limited to the above-described embodiments, and those skilled in the art will be within the scope of the appended claims. It is possible to perform various deformations that can be assumed.

  For example, in the above-described embodiment, the case has been described where the learning unit 13 and the determination unit 14 are mounted on the same computer. However, the learning unit 13 and the determination unit 14 are mounted on separate computers. Is also good.

  1, 1A, 1B: Image inspection support device, 10: Image acquisition unit, 11: Preprocessing unit, 12: Storage unit, 13: Learning unit, 14: Judgment unit, 130: First feature amount extraction unit, 131: Image Restoring unit, 132 error calculating unit, 133 error back-propagating unit, 140 second feature amount extracting unit, 141 similarity calculating unit, 142 analyzing unit, 143 result output unit, 101 bus, 102 arithmetic Device, 103 CPU, 104 main storage device, 105 communication control device, 106 external storage device, 107 display device, 108 input device, 106a image data storage unit, 106b setting information storage unit, 106c Learning information storage unit, 106d: determination information storage unit, 106e: program storage unit.

Claims (7)

An image inspection support device that supports confirmation of a label attached to an image used in learning of a learning type image inspection device,
An image acquisition unit that acquires an image used for learning and determination,
A learning unit that performs learning of the neural network based on the acquired learning image and constructs a learned neural network that outputs a feature amount that can restore the learning image;
Using the learned neural network, the feature amount of the acquired determination image is extracted and calculated from the feature amount, and the target determination image and each of the other determination images are calculated. A determination unit that determines whether or not confirmation is necessary to correct the label attached to the target determination image based on the index values indicating the mutual correlation, and outputs a determination result.
The acquired image for determination includes a label given in advance. The image inspection support apparatus. The image inspection support device according to claim 1,
The learning unit includes:
A first feature amount extraction unit that performs a convolution layer operation with the learning image as an input and extracts a feature amount of the learning image;
With the feature amount extracted by the first feature amount extraction unit as input, a deconvolution layer having an inverse structure corresponding to the convolution layer is calculated, and a restored image in which the learning image is restored is output. An image restoration unit;
An error calculation unit that calculates an error between the learning image input to the first feature amount extraction unit and the restored image,
An error back propagation unit that learns the convolution layer and the deconvolution layer using an error back propagation method so that the calculated error is smaller,
With
The determination unit includes:
A second feature value extraction unit that performs an operation on a learned convolutional layer obtained by learning by the learning unit and receives a feature value of the determination image, using the determination image as an input;
A similarity calculation unit that calculates the index value based on the feature amount extracted by the second feature amount extraction unit;
From the index value, the other determination image similar to the target determination image exists, and the label attached to the similar another determination image and the target When the label attached to the image for determination is different from each other, an analysis unit that determines that the label attached to the image for determination as the target needs confirmation for correction,
A result output unit that outputs a determination result by the analysis unit,
An image inspection support device, comprising: The image inspection support device according to claim 2,
The analysis unit, when there is no other image for determination similar to the image for determination as the target, confirms a label attached to the image for determination as the target for correction. An image inspection support apparatus characterized in that it is determined that the image inspection may be performed. The image inspection support device according to claim 2 or 3,
The analysis unit includes:
A similarity storage unit that stores the index value calculated by the similarity calculation unit for each image for determination,
A threshold value setting unit that sets a threshold value for the index value;
Based on the index value stored by the similarity storage unit, the threshold value, and the label attached to the image for determination, modify the label attached to the image for determination as the target. A labeling determination unit that determines whether confirmation is necessary
With
The similarity storage unit uses the index value for each of the images for determination, and uses the index value to determine a predetermined number of images in order from the other image for determination having higher similarity with the image for determination to be targeted. Selecting, saving the selected other image for determination together with a label,
The labeling determination unit compares the index value calculated for the target determination image with the threshold value, and attaches the label to the target determination image based on the comparison result. An image inspection support device, which determines whether or not confirmation for correcting a label is required. The image inspection support device according to claim 4,
The labeling determination unit, when the label attached to the target image for determination and the label attached to the other determination image stored in the similarity storage unit are different from each other An image inspection support apparatus for determining that a label attached to the target determination image needs confirmation for correction. The image inspection support device according to any one of claims 2 to 5,
The result output unit includes:
A feature amount compression unit that compresses a dimension of the feature amount extracted by the second feature amount extraction unit;
A feature amount visualization unit that generates a scatter diagram for each image for determination from the compressed feature amount and displays the scatter diagram on a display device,
An operation unit configured to externally receive a label correction instruction for the target determination image displayed on the display device,
A data correction unit that corrects a label attached to the target image for determination based on the correction instruction and stores the corrected data in a storage unit. An image inspection support method for supporting confirmation of a label attached to an image used in learning of a learning-type image inspection device,
An image acquisition step of acquiring an image used for learning and determination;
Learning step of learning the neural network based on the acquired learning image, and constructing a learned neural network that outputs a feature amount that can restore the learning image,
Using the learned neural network, extract the feature amount of the acquired determination image, calculated from the extracted feature amount, each of the target determination image and the other determination image A determination step of determining whether or not confirmation for correcting a label attached to the target determination image is necessary and outputting a determination result based on an index value indicating a mutual correlation with ,
The acquired image for determination includes a label assigned in advance, The image inspection support method.