JP7070308B2 - Estimator generator, inspection device, estimator generator method, and estimator generator - Google Patents

Description

The present invention relates to an estimator generation device, an inspection device, an estimator generation method, and an estimator generation program.

Conventionally, in the scene of manufacturing a product such as a production line, a technique of photographing the manufactured product with an imaging device and inspecting the quality of the product based on the obtained image data has been used. For example, in Patent Document 1, it is determined whether the inspection target imaged in the image is normal or abnormal based on the learned first neural network, and when it is determined that the inspection target is abnormal, it is determined. An inspection device that classifies the type of the anomaly based on the learned second neural network has been proposed.

Japanese Unexamined Patent Publication No. 2012-026892 Japanese Unexamined Patent Publication No. 2016-11290

The inventors of the present invention have the following problems in the conventional technique of determining the quality of a product from image data by using an estimator composed of a learning model such as a neural network as in Patent Document 1. I found that it could happen. That is, the larger the size of the learning image, the more complicated the configuration of the estimator for determining the quality of the product. Therefore, the calculation cost of the estimator becomes large, a memory shortage occurs during the calculation process of machine learning, the calculation process takes a long time, and the calculation process is not completed within a predetermined time. There may be a problem that problems such as the above occur.

Therefore, the inventors of the present invention examined to construct an estimator that extracts features from the learning image and judges the quality of the product based on the extracted features. For example, Patent Document 2 proposes an inspection system that calculates a feature amount of image data obtained by imaging an object and detects defects in the appearance of the object based on the calculated feature amount. Since the features extracted from the training image are partially reduced in information by the extraction process, the amount of information is usually smaller (that is, lower dimensional) than the training image itself. Therefore, the configuration of the estimator using this feature can be simpler than that of the estimator using the learning image itself, and thereby the calculation cost of the estimator can be reduced. Therefore, according to this method, it is possible to shorten the learning time of the estimator and save the hardware resources used for the machine learning of the estimator.

However, the present inventors have found that this method has the following problems. That is, the accuracy of the determination by the estimator depends on the number of training data. Therefore, if the number of training images is small, the number of features that can be obtained is also small, and the accuracy of defect estimation by the trained learning model (estimator) becomes insufficient. In particular, the feature extracted from the training image is that the information contained in the training image is partially reduced by the extraction process. That is, the amount of information that the estimator can obtain from this feature is less than the amount of information that the estimator can obtain from the training image itself. Therefore, the accuracy of defect estimation of the estimator constructed by this method tends to be worse than that of the estimator constructed by using the learning image. Therefore, when the number of training data is small, the accuracy of defect estimation by the trained learning model (estimator) may become extremely poor.

It should be noted that such a problem is not only in the case of constructing an estimator for judging the quality of a product by machine learning, but also in any situation in which an estimator for estimating some attribute of an object reflected in an image is constructed. Can occur in. For example, when constructing an estimator for detecting the facial organs of a subject, if the number of facial images (learning images) used as learning data is small, the number of features obtained from the facial images is also small. Therefore, the accuracy of organ estimation by the estimator may be insufficient.

The present invention has been made in view of such a situation on one aspect, and the purpose of the present invention is to reduce the calculation cost of the estimator and to capture the image even if the number of learning images is relatively small. It is to provide a technique for constructing an estimator capable of estimating the attributes of an object with high accuracy.

The present invention adopts the following configuration in order to solve the above-mentioned problems.

That is, the estimator generator according to one aspect of the present invention is composed of a combination of a learning image showing a product to be visually inspected and a label indicating the type of defect contained in the product. A data acquisition unit for acquiring a data set, an extraction unit for extracting one or a plurality of first features from the training image included in each learning data set, and an expansion process for the extracted one or a plurality of first features. By applying the above-mentioned one or a plurality of first features and the above-mentioned one or a plurality of first features and the above-mentioned 1 by performing machine learning with an enlargement processing unit that generates one or a plurality of second features different from the above-mentioned one or a plurality of first features. Or a learning process that builds an estimator trained to estimate the type of defect contained in the product in the training image so as to match the corresponding label from at least one of the plurality of second features. It is equipped with a department.

In this configuration, the estimator is not trained to estimate the quality of the product directly from the training image, but to estimate the quality of the product from the features extracted from the training image. In general, the dimension of the extracted feature is lower than that of the training image itself, so that the amount of information contained in the extracted feature is smaller than that of the training image. Therefore, in the case of training to estimate the quality of the product from the extracted features, the amount of information of the input data is compared with the case of training to estimate the quality of the product directly from the training image. Since it can be suppressed, the configuration of the estimator can be simplified. In addition, in this configuration, by applying the enlargement processing to the first feature extracted from the learning image, a second feature different from the first feature is generated. This makes it possible to increase the variety of features. That is, the number of training data used for machine learning can be increased. Therefore, even if the number of training images is small, it is possible to prepare a sufficient number of training data to ensure a certain accuracy of defect estimation. Therefore, according to the configuration, it is possible to construct an estimator capable of estimating the attributes of the object reflected in the image with high accuracy even if the number of learning images is relatively small, while reducing the calculation cost of the estimator. Can be done.

The type and data format of the feature may not be particularly limited as long as it is some information that can be derived from the image, and may be appropriately selected according to the embodiment. For feature extraction, for example, general image processing (for example, edge extraction), SIFT (Scale-Invariant Feature Transform), trained learning model (for example, convolutional neural network) and the like may be used. For example, a feature map may be derived from the training image as the first feature by using a convolutional neural network. The augmentation process (data augmentation) is a process of deriving different features from one or more features. For the enlargement process, for example, a process of synthesizing a plurality of features, a process of converting features by geometric conversion, or the like may be used. When a feature is derived as an image, the process of converting the feature includes a process of synthesizing predetermined noise (white noise, random noise, etc.) into the feature image, a process of converting pixels included in the feature image, and the like. It's okay. Pixel conversion may include color conversion, density conversion, and the like, and a predetermined conversion function may be used for these conversions.

Further, the type of the product is not particularly limited as long as it can be subject to visual inspection, and may be appropriately selected according to the embodiment. The product may be, for example, a product transported on a production line for electronic devices, electronic parts, automobile parts, chemicals, foods, and the like. The electronic device is, for example, a mobile phone or the like. Electronic components are, for example, boards, chip capacitors, liquid crystals, relay windings, and the like. Automotive parts are, for example, connecting rods, shafts, engine blocks, power window switches, panels and the like. The chemical may be, for example, a packaged tablet, an unpackaged tablet, or the like. Further, the type of the defect may be not particularly limited as long as it may be related to the quality of the product, and may be appropriately selected according to the embodiment. Defects include, for example, scratches, stains, cracks, dents, burrs, color unevenness, foreign matter contamination, and the like.

In the estimator generator according to the one aspect, the one or a plurality of first features may be extracted as feature images smaller in size than the learning image, respectively. According to this configuration, features can be obtained in a format suitable for enlargement processing, so that the number of training data (features) used for machine learning can be increased. This makes it possible to construct an estimator capable of estimating an object reflected in an image with high accuracy. The feature image is, for example, a feature map derived by a convolutional neural network.

In the estimator generation device according to the one aspect, weights may be set for each of the plurality of first features, and applying the enlargement processing means that the set weights are applied to each of the plurality of first features. It may include subjecting to and synthesizing the plurality of weighted first features. According to this configuration, it is possible to appropriately enrich the variation of the features used for machine learning, and thereby it is possible to construct an estimator capable of estimating the object to be reflected in the image with high accuracy.

In the estimator generator according to one aspect, applying the enlargement process may include applying a geometric transformation to the one or more first features. According to this configuration, it is possible to appropriately enrich the variation of the features used for machine learning, and thereby it is possible to construct an estimator capable of estimating the object to be reflected in the image with high accuracy.

In the estimator generator according to one aspect, the geometric transformation may be translation, rotational movement, inversion, or a combination thereof. According to this configuration, it is possible to appropriately enrich the variation of the features used for machine learning, and thereby it is possible to construct an estimator capable of estimating the object to be reflected in the image with high accuracy.

Further, the inspection device according to one aspect of the present invention includes an image acquisition unit that acquires a target image of a product to be visually inspected, an extraction unit that extracts features from the acquired target image, and any of the above. An inspection unit that inspects whether or not the product shown in the target image contains a defect based on the extracted features by using the estimator constructed by the estimator generator according to the embodiment of the above. It is provided with an output unit that outputs information about the inspection result. According to this configuration, the appearance inspection of the product can be performed with high accuracy.

Further, the estimator generation device and the inspection device according to each of the above forms estimate not only the scene of constructing an estimator for judging the quality of a product from an image by machine learning, but also some attribute of an object reflected in the image. It may be applied in any situation to build an estimator for. For example, the estimator generation device and the inspection device according to each of the above forms may be applied to a scene of constructing an estimator for detecting an organ from a face image of a subject's face. Further, the estimator generator according to each of the above embodiments may be configured to omit the configuration of extracting the first feature from the learning image and directly acquire the first feature.

For example, the estimator generator according to one aspect of the present invention is a data acquisition unit that acquires a plurality of learning data sets each composed of a combination of a learning image showing an object and a label indicating the attribute of the object. By applying the enlargement processing to the extraction unit that extracts one or a plurality of first features from the learning image included in each learning data set and the extracted one or a plurality of first features, the one or a plurality of the first features are extracted. At least one of the one or more first features and the one or more second features by performing machine learning with an enlargement processing unit that generates one or more second features different from the plurality of first features. From this, it is provided with a learning processing unit that constructs an estimator trained to estimate the attributes of the object reflected in the learning image so as to match the corresponding label.

Further, for example, the estimator generator according to one aspect of the present invention is configured by a combination of one or a plurality of first features extracted from a learning image of an object and a label indicating the attribute of the object. By applying the expansion process to the one or more first features included in each of the learning data sets and the data acquisition unit for acquiring the plurality of learning data sets, the one or more first features are An enlargement processing unit that generates one or a plurality of different second features, and a corresponding label from at least one of the one or more first features and the one or more second features by performing machine learning. A learning processing unit that constructs an estimator trained to estimate the attributes of the object reflected in the learning image so as to match is provided.

Further, for example, the estimation device according to one aspect of the present invention has an image acquisition unit for acquiring an object image in which an object is captured, an extraction unit for extracting features from the acquired target image, and any of the above forms. Using the estimator constructed by the estimator generator, the estimator for estimating the state of the object reflected in the target image and information on the estimation result are output based on the extracted features. It has an output unit.

The type of the "object" is not particularly limited as long as it can be an object for which some attribute is estimated, and may be appropriately selected according to the embodiment. The object may be, for example, a product, a person, a body part of the person (for example, a face, etc.) to be subject to the visual inspection. Further, the "attribute" of the object to be estimated may be any property related to any property of the object, and may include all kinds of attributes of the object that can be estimated by the estimator. When the object is a person's face, the attributes to be judged are, for example, the type of facial expression, the position of facial parts (including organs) (including the relative positional relationship between specific organs), and facial parts. It may be the shape of the face, the color of the facial part, the state of the facial part (opening, angle, etc.), the attribute of the individual who owns the face, and the like. When the object is the body of a person, the attribute to be determined may be, for example, a pose of the body. When the object is a work to be worked on, the attributes to be determined may be, for example, the position and posture of the work.

The inspection system according to one aspect of the present invention may be composed of an estimator generator and an inspection apparatus according to any one of the above embodiments. Further, the estimation system according to one aspect of the present invention may be composed of an estimator generator and an estimation device according to any one of the above embodiments. Further, as another form of the estimator generator, the inspection device, the inspection system, and the estimation system according to each of the above embodiments, one aspect of the present invention may be an information processing method that realizes each of the above configurations. , A program, or a storage medium that stores such a program and can be read by a computer or the like. Here, the storage medium that can be read by a computer or the like is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

For example, in the estimator generation method according to one aspect of the present invention, a computer is configured by a combination of a learning image showing a product to be visually inspected and a label indicating the type of defect contained in the product. A step of acquiring a plurality of training data sets, a step of extracting one or a plurality of first features from the training images included in each training data set, and an expansion process to the extracted one or a plurality of first features. By applying, the step of generating one or more second features different from the one or more first features, and by performing machine learning, the one or more first features and the one or more From at least one of the second features of the above, a step of constructing an estimator trained to estimate the type of defect contained in the product in the training image so as to match the corresponding label. It is an information processing method to be executed.

Further, for example, the estimator generation program according to one aspect of the present invention is configured by a combination of a learning image showing a product to be visually inspected and a label indicating the type of a defect contained in the product on a computer. The step of acquiring the plurality of learning data sets, the step of extracting one or a plurality of first features from the learning images included in each learning data set, and the extracted one or a plurality of first features. By applying the enlargement processing, a step of generating one or a plurality of second features different from the one or a plurality of first features, and by performing machine learning, the one or a plurality of first features and the above 1 Or with the step of constructing an estimator trained to estimate the type of defect contained in the product in the training image so as to match the corresponding label from at least one of the plurality of second features. , Is a program for executing.

According to the present invention, it is possible to construct an estimator capable of estimating the attributes of an object appearing in an image with high accuracy even if the number of learning images is relatively small, while reducing the calculation cost of the estimator. ..

FIG. 1 schematically illustrates an example of a situation in which the present invention is applied. FIG. 2 schematically illustrates an example of the hardware configuration of the estimator generator according to the embodiment. FIG. 3 schematically illustrates an example of the hardware configuration of the inspection device according to the embodiment. FIG. 4 schematically illustrates an example of the software configuration of the estimator generator according to the embodiment. FIG. 5 schematically illustrates an example of the software configuration of the inspection device according to the embodiment. FIG. 6 illustrates an example of the processing procedure of the estimator generator according to the embodiment. FIG. 7 schematically illustrates an example of the feature extraction process according to the embodiment. FIG. 8A schematically illustrates an example (weighted synthesis) of the enlargement processing according to the present embodiment. FIG. 8B schematically illustrates an example of a scene dealing with a first feature having a different size (scale). FIG. 9 schematically illustrates an example (geometric transformation) of the enlargement processing according to the present embodiment. FIG. 10 illustrates an example of the processing procedure of the inspection device according to the embodiment. FIG. 11 schematically illustrates an example of the software configuration of the estimator generator according to the modified example. FIG. 12 schematically illustrates another example of a situation in which the present invention is applied. FIG. 13 schematically illustrates an example of the software configuration of the estimation device according to the modified example.

Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. The data appearing in the present embodiment are described in natural language, but more specifically, they are specified in a pseudo language, a command, a parameter, a machine language, etc. that can be recognized by a computer.

§1 Application example First, an example of a situation in which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates an example of an application scene of the inspection system 100 according to the present embodiment. In the example of FIG. 1, the present invention is applied to a scene where a visual inspection of a product R manufactured on a production line is performed. The product R subject to the visual inspection is an example of the "object" of the present invention, and the presence or absence of a defect and the type of the defect of the product R are an example of the "attribute of the object" of the present invention. However, the object of application of the present invention does not have to be limited to such an example, and can be applied to any situation in which some attribute of an object is estimated from an image.

As shown in FIG. 1, the inspection system 100 according to the present embodiment includes an estimator generator 1 and an inspection device 2 connected to each other via a network, and determines the presence or absence of defects and the types of defects in the product R. An estimator for estimation is generated, and the generated estimator is configured to perform a visual inspection of the product R. The type of network between the estimator generator 1 and the inspection device 2 may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, and the like.

The estimator generator 1 according to the present embodiment is a computer configured to construct an estimator 50 for estimating the presence / absence of a defect and the type of defect of the product R from an image by performing machine learning. be. Specifically, the estimator generator 1 has a plurality of learnings configured by a combination of a learning image 61 showing the product R to be visually inspected and a label 63 indicating the type of defect included in the product R. Acquire the data set 60. Next, the estimator generator 1 extracts one or a plurality of first features 71 from the training image 61 included in each training data set 60, and applies an enlargement process to the extracted one or a plurality of first features 71. By doing so, one or more second features 73 different from one or more first features 71 are generated.

Then, the estimator generator 1 learns from at least one of one or a plurality of first features 71 and one or a plurality of second features 73 so as to match the corresponding label 63 by performing machine learning. An estimator 50 trained to estimate the type of defect contained in the product R shown in image 61 is constructed. That is, the estimator generator 1 uses one or more first features 71 and one or more second features 73 as training data (input data), and uses the corresponding label 63 as correct answer data (teacher data). Then, machine learning of the estimator 50 is carried out. As a result, the estimator generator 1 inputs at least one of one or a plurality of first features 71 and one or a plurality of second features 73 derived from the training image 61 for each training data set 60. An estimator 50 trained to output an output value corresponding to the type of defect indicated by the label 63 associated with the training image 61 is constructed.

On the other hand, the inspection device 2 according to the present embodiment is a computer configured to perform a visual inspection of the product R by using the trained estimator 50 constructed by the estimator generation device 1. be. The inspection device 2 is an example of an estimation device for estimating some attribute of an object shown in an image. Specifically, the inspection device 2 acquires a target image 81 showing the product R to be visually inspected. In the present embodiment, the camera CA is connected to the inspection device 2. Therefore, the inspection device 2 acquires the target image 81 by photographing the product R with this camera CA.

Next, the inspection device 2 extracts one or a plurality of features 83 from the acquired target image 81. Subsequently, the inspection device 2 utilizes the trained estimator 50 constructed by the estimator generation device 1, and the product R shown in the target image 81 is defective based on the extracted one or a plurality of features 83. Check if it is included. That is, the inspection device 2 inputs the extracted one or a plurality of features 83 into the trained estimator 50, and executes the arithmetic processing of the trained estimator 50. As a result, the inspection device 2 acquires an output value corresponding to the result of inspecting (estimating) whether or not the product R contains a defect from the trained estimator 50. Then, the inspection device 2 outputs information regarding the inspection result.

As described above, in the present embodiment, the estimator 50 is not trained to directly estimate the quality of the product R from the learning image 61, but the first feature 71 and the first feature 71 derived from the learning image 61. 2 Trained to estimate the quality of product R from at least one of feature 73. Since the first feature 71 and the second feature 73 are derived so as to have a lower dimension than the learning image 61 itself, the amount of information possessed by the first feature 71 and the second feature 73 is larger than the amount of information of the learning image 61. Less. Therefore, as compared with the case where the training image 61 is trained to directly estimate the quality of the product R, in the present embodiment, the amount of information of the input data for the estimator 50 can be suppressed, so that the estimator can be suppressed. The configuration of 50 can be simplified.

In addition, in the present embodiment, by applying the enlargement processing to the first feature 71 extracted from the learning image 61, the second feature 73 different from the first feature 71 is generated. As a result, it is possible to increase the variation of features for deriving the quality of the product R. That is, the number of training data used for machine learning can be increased. Therefore, even if the number of training images 61 is small, it is possible to prepare a sufficient number of training data to ensure a certain accuracy of defect estimation. Therefore, according to the estimator generator 1 according to the present embodiment, while reducing the calculation cost of the estimator 50, even if the number of learning images 61 is relatively small, the quality of the product R reflected in the image is highly accurate. It is possible to construct an estimator 50 that can be estimated. Further, according to the inspection device 2 according to the present embodiment, the appearance inspection of the product R can be performed with high accuracy and high speed by using the estimator 50.

In the example of FIG. 1, the estimator generator 1 and the inspection device 2 are separate computers. However, the configuration of the inspection system 100 does not have to be limited to such an example. The estimator generator 1 and the inspection device 2 may be configured by an integrated computer. Further, the estimator generation device 1 and the inspection device 2 may each be composed of a plurality of computers. Further, the estimator generator 1 and the inspection device 2 do not have to be connected to the network. In this case, the exchange of data between the estimator generation device 1 and the inspection device 2 may be performed via a storage medium such as a non-volatile memory.

The type and data format of each feature (71, 73) is not particularly limited as long as it is some information that can be derived from an image, and may be appropriately selected depending on the embodiment. For the extraction of the first feature 71, for example, general image processing (for example, edge extraction), SIFT, a trained learning model (for example, a convolutional neural network) or the like may be used. The enlargement process is a process of deriving a second feature 73 different from one or a plurality of first features 71. For the enlargement process, for example, a process of synthesizing a plurality of first features 71, a process of converting the first feature 71 by geometric conversion, or the like may be used.

Further, the type of the product R may not be particularly limited as long as it can be a target of visual inspection, and may be appropriately selected according to the embodiment. The product R may be, for example, a product transported on a production line for electronic devices, electronic parts, automobile parts, chemicals, foods, and the like. The electronic device is, for example, a mobile phone or the like. Electronic components are, for example, boards, chip capacitors, liquid crystals, relay windings, and the like. Automotive parts are, for example, connecting rods, shafts, engine blocks, power window switches, panels and the like. The chemical may be, for example, a packaged tablet, an unpackaged tablet, or the like. Further, the type of the defect may not be particularly limited as long as it may be related to the quality of the product R, and may be appropriately selected according to the embodiment. Defects include, for example, scratches, stains, cracks, dents, burrs, color unevenness, foreign matter contamination, and the like. The type of defect indicated by label 63 may also include the absence of defects in product R.

§2 Configuration example [Hardware configuration]
<Estimator generator>
Next, an example of the hardware configuration of the estimator generator 1 according to the present embodiment will be described with reference to FIG. FIG. 2 schematically illustrates an example of the hardware of the estimator generator 1 according to the present embodiment.

As shown in FIG. 2, the estimator generator 1 according to the present embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected. Is. In FIG. 2, the communication interface is described as "communication I / F".

The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, which are hardware processors, and is configured to execute information processing based on a program and various data. To. The storage unit 12 is an example of a memory, and is composed of, for example, a hard disk drive, a solid state drive, or the like. In the present embodiment, the storage unit 12 stores various information such as the estimator generation program 121, the plurality of learning data sets 60, and the learning result data 125.

The estimator generation program 121 is a program for causing the estimator generation device 1 to execute machine learning information processing (FIG. 6), which will be described later, to construct a learned estimator 50. The estimator generation program 121 includes a series of instructions for this information processing. Each training data set 60 is composed of the training image 61 and the label 63, and is used for machine learning of the estimator 50. The learning result data 125 is data for setting the trained estimator 50 constructed by machine learning. The learning result data 125 is generated as an execution result of the estimator generation program 121. Details will be described later.

The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 13, the estimator generation device 1 can perform data communication via a network with another information processing device (for example, an inspection device 2).

The input device 14 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 15 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the estimator generator 1 by using the input device 14 and the output device 15.

The drive 16 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 91. The type of the drive 16 may be appropriately selected according to the type of the storage medium 91. At least one of the estimator generation program 121 and the plurality of training data sets 60 may be stored in the storage medium 91.

The storage medium 91 transfers the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the computer or other device, the machine or the like can read the information of the recorded program or the like. It is a medium to accumulate. The estimator generation device 1 may acquire at least one of the estimator generation program 121 and the plurality of training data sets 60 from the storage medium 91.

Here, in FIG. 2, as an example of the storage medium 91, a disc-type storage medium such as a CD or a DVD is illustrated. However, the type of the storage medium 91 is not limited to the disc type, and may be other than the disc type. Examples of storage media other than the disk type include semiconductor memories such as flash memories.

Regarding the specific hardware configuration of the estimator generator 1, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 11 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), or the like. The storage unit 12 may be composed of a RAM and a ROM included in the control unit 11. At least one of the communication interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The estimator generator 1 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the estimator generator 1 may be a general-purpose server device, a PC (Personal Computer), or the like, in addition to an information processing device designed exclusively for the provided service.

<Inspection equipment>
Next, an example of the hardware configuration of the inspection device 2 according to the present embodiment will be described with reference to FIG. FIG. 3 schematically illustrates an example of the hardware configuration of the inspection device 2 according to the present embodiment.

As shown in FIG. 3, in the inspection device 2 according to the present embodiment, the control unit 21, the storage unit 22, the communication interface 23, the input device 24, the output device 25, the drive 26, and the external interface 27 are electrically connected. It is a computer. In FIG. 3, the external interface is described as "external I / F". The control units 21 to drive 26 of the inspection device 2 may be configured in the same manner as the control units 11 to drive 16 of the estimator generator 1, respectively.

That is, the control unit 21 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 22 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 22 stores various information such as the inspection program 221 and the learning result data 125.

The inspection program 221 is a program for causing the inspection device 2 to execute information processing (FIG. 10), which will be described later, for performing a visual inspection of the product R shown in the target image 81 by using the trained estimator 50. The inspection program 221 includes a series of instructions for this information processing. Details will be described later.

The communication interface 23 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 23, the inspection device 2 can perform data communication via a network with another information processing device (for example, an estimator generation device 1).

The input device 24 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 25 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the inspection device 2 by using the input device 24 and the output device 25.

The drive 26 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 92. At least one of the inspection program 221 and the learning result data 125 may be stored in the storage medium 92. Further, the inspection device 2 may acquire at least one of the inspection program 221 and the learning result data 125 from the storage medium 92.

The external interface 27 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of the external interfaces 27 may be appropriately selected according to the type and number of connected external devices. In the present embodiment, the inspection device 2 is connected to the camera CA via the external interface 27.

The camera CA is used to acquire the target image 81 in which the product R is captured. The type and location of the camera CA are not particularly limited and may be appropriately determined according to the embodiment. As the camera CA, for example, a known camera such as a digital camera or a video camera may be used. Further, the camera CA may be arranged in the vicinity of the production line on which the product R is conveyed. When the camera CA includes a communication interface, the inspection device 4 may be connected to the camera CA via the communication interface 23 instead of the external interface 27.

Regarding the specific hardware configuration of the inspection device 2, the components can be omitted, replaced, or added as appropriate according to the embodiment, as in the case of the estimator generation device 1. For example, the control unit 21 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, DSP and the like. The storage unit 22 may be composed of a RAM and a ROM included in the control unit 21. At least one of the communication interface 23, the input device 24, the output device 25, the drive 26, and the external interface 27 may be omitted. The inspection device 2 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, as the inspection device 2, in addition to an information processing device designed exclusively for the provided service, a general-purpose server device, a general-purpose desktop PC, a notebook PC, a tablet PC, a mobile phone including a smartphone, or the like may be used.

[Software configuration]
<Estimator generator>
Next, an example of the software configuration of the estimator generator 1 according to the present embodiment will be described with reference to FIG. FIG. 4 schematically illustrates an example of the software configuration of the estimator generator 1 according to the present embodiment.

The control unit 11 of the estimator generation device 1 expands the estimator generation program 121 stored in the storage unit 12 into the RAM. Then, the control unit 11 interprets and executes the estimator generation program 121 expanded in the RAM by the CPU, and controls each component. As a result, as shown in FIG. 4, the estimator generator 1 according to the present embodiment includes the data acquisition unit 111, the extraction unit 112, the enlargement processing unit 113, the learning processing unit 114, and the storage processing unit 115 as software modules. Operates as a equipped computer. That is, in the present embodiment, each software module of the estimator generator 1 is realized by the control unit 11 (CPU).

The data acquisition unit 111 acquires a plurality of learning data sets 60 each composed of a combination of a learning image 61 showing the product R to be visually inspected and a label 63 indicating the type of defect included in the product R. .. The extraction unit 112 extracts one or a plurality of first features 71 from the training image 61 included in each training data set 60. The enlargement processing unit 113 applies the enlargement processing to the extracted one or a plurality of first features 71 to generate one or a plurality of second features 73 different from the one or a plurality of first features 71.

The learning processing unit 114 holds an estimator 50 for performing machine learning. The learning processing unit 114 performs machine learning on the training image 61 so as to match the corresponding label 63 from at least one of the one or a plurality of first features 71 and the one or a plurality of second features 73. An estimator 50 trained to estimate the type of defect contained in the imaged product R is constructed. The storage processing unit 115 stores information about the constructed learned estimator 50 in the storage unit 12 as learning result data 125.

In addition, "estimation" may be either, for example, deriving a discrete value (class) by grouping (classification, identification) or deriving a continuous value by regression. The data format of the label 63 may be appropriately selected according to the form indicating the type of defect that may be included in the product R (that is, the output format of the estimator 50). When the type of defect is expressed by a discrete value such as a class, the label 63 may be composed of numerical data of the discrete value. Alternatively, when the type of defect is expressed by a continuous value such as the probability that a defect of the corresponding type exists, the label 63 may be composed of numerical data of the continuous value.

(Estimator)
Next, an example of the configuration of the estimator 50 will be described. As shown in FIG. 4, the estimator 50 according to the present embodiment is configured by a neural network. Specifically, the estimator 50 is composed of a multi-layered neural network used for so-called deep learning, and includes an input layer 501, an intermediate layer (hidden layer) 503, and an output layer 505.

In the example of FIG. 4, the neural network constituting the estimator 50 includes one intermediate layer 503, the output of the input layer 501 is input to the intermediate layer 503, and the output of the intermediate layer 503 is the output layer. It is input to 505. However, the configuration of the estimator 50 does not have to be limited to such an example. The number of intermediate layers 503 does not have to be limited to one. The estimator 50 may include two or more intermediate layers 503.

Each layer (501, 503, 505) comprises one or more neurons. The number of neurons contained in each layer (501, 503, 505) may be appropriately set according to the embodiment. Neurons in adjacent layers are appropriately connected to each other, and a weight (bonding load) is set for each connection. In the example of FIG. 4, each neuron is connected to all neurons in adjacent layers. However, the connection of neurons does not have to be limited to such an example, and may be appropriately set according to the embodiment.

A threshold is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold. That is, the weight of the connection between each of these neurons and the threshold value of each neuron are examples of the parameters of the estimator 50 used for arithmetic processing. The learning processing unit 114 inputs at least one of one or a plurality of first features 71 and one or a plurality of second features 73 to the input layer 501, and uses these parameters to perform arithmetic processing of the estimator 50. Run.

As a result of this arithmetic processing, the learning processing unit 114 acquires an output value corresponding to the result of estimating the quality (that is, the type of defect) of the product R shown in the learning image 61 from the output layer 505. Subsequently, the learning processing unit 114 calculates an error between the acquired output value and the value of the label 63. Then, the learning processing unit 114 adjusts the value of the parameter of the estimator 50 so that the sum of the calculated errors becomes small. The learning processing unit 114 repeats adjusting the value of the parameter of the estimator 50 until the sum of the errors between the output value obtained from the output layer 505 and the value of the label 63 becomes equal to or less than the threshold value. As a result, when the learning processing unit 114 inputs at least one of one or a plurality of first features 71 and one or a plurality of second features 73 derived from the learning image 61 for each learning data set 60, the learning is performed. Construct an estimator 50 trained to output an output value corresponding to the type of defect indicated by the label 63 associated with the image 61.

After the processing of this machine learning is completed, the preservation processing unit 115 determines the configuration of the constructed trained estimator 50 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, and each neuron. Information indicating the transmission function) and the arithmetic parameters (for example, the weight of the connection between each neuron, the threshold value of each neuron) is generated as the learning result data 125. Then, the storage processing unit 115 stores the generated learning result data 125 in the storage unit 12.

<Inspection equipment>
Next, an example of the software configuration of the inspection device 2 according to the present embodiment will be described with reference to FIG. FIG. 5 schematically illustrates an example of the software configuration of the inspection device 2 according to the present embodiment.

The control unit 21 of the inspection device 2 expands the inspection program 221 stored in the storage unit 22 into the RAM. Then, the control unit 21 interprets and executes the inspection program 221 expanded in the RAM by the CPU, and controls each component. As a result, as shown in FIG. 5, the inspection device 2 according to the present embodiment operates as a computer including an image acquisition unit 211, an extraction unit 212, an inspection unit 213, and an output unit 214 as software modules. That is, in the present embodiment, each software module of the inspection device 2 is also realized by the control unit 21 (CPU) in the same manner as the estimator generation device 1.

The image acquisition unit 211 acquires the target image 81 that captures the product R that is the target of the visual inspection. For example, the image acquisition unit 211 acquires the target image 81 by photographing the product R with the camera CA. The extraction unit 212 extracts one or a plurality of features 83 from the acquired target image 81. The inspection unit 213 includes the trained estimator 50 constructed by the estimator generator 1 by holding the learning result data 125. Specifically, the inspection unit 213 sets the trained estimator 50 with reference to the learning result data 125. Then, the inspection unit 213 inputs the extracted one or a plurality of features 83 into the trained estimator 50, and executes the arithmetic processing of the trained estimator 50 to determine whether or not the product R contains a defect. The output value corresponding to the result of the inspection (that is, the quality of the product R is determined) is acquired from the trained estimator 50. As a result, the inspection unit 213 inspects whether or not the product R shown in the target image 81 contains a defect based on the extracted one or a plurality of features 83 by using the trained estimator 50. The output unit 214 outputs information regarding the result of the inspection.

<Others>
Each software module of the estimator generator 1 and the inspection device 2 will be described in detail in an operation example described later. In this embodiment, an example in which each software module of the estimator generator 1 and the inspection device 2 is realized by a general-purpose CPU is described. However, some or all of the above software modules may be implemented by one or more dedicated processors. Further, with respect to the software configurations of the estimator generation device 1 and the inspection device 2, software modules may be omitted, replaced, or added as appropriate according to the embodiment.

§3 Operation example [Estimator generator]
Next, an operation example of the estimator generator 1 according to the present embodiment will be described with reference to FIG. FIG. 6 shows an example of the processing procedure of the estimator generator 1 according to the present embodiment. The processing procedure described below is an example of the "estimator generation method" of the present invention. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S101)
In step S101, the control unit 11 operates as a data acquisition unit 111, and acquires a plurality of learning data sets 60 each composed of a combination of the learning image 61 and the label 63.

The method for acquiring the plurality of training data sets 60 does not have to be particularly limited, and may be appropriately selected depending on the embodiment. For example, a camera and a product R are prepared, and a defective or non-defective product R is photographed by the camera. As a result, the learning image 61 can be generated. Then, a label 63 indicating the type of defect (including no defect) included in the product R reflected in the learning image 61 is appropriately generated, and the generated label 63 is associated with the corresponding learning image 61. This makes it possible to generate each training data set 60.

The generation of the training data set 60 may be automatically performed by the operation of the computer, or may be manually performed by the operation of the operator. Further, the information processing for generating the learning data set 60 may be executed in the estimator generator 1 or may be executed in a computer other than the estimator generator 1.

When the estimator generator 1 generates each learning data set 60, the control unit 11 acquires a plurality of learning data sets 60 by automatically or manually executing the information processing by the operation of the operator. On the other hand, when each learning data set 60 is generated by another computer, the control unit 11 acquires a plurality of learning data sets 60 generated by the other computer via, for example, a network, a storage medium 91, or the like.

As described above, the data format of the label 63 may be appropriately selected according to the output format of the estimator 50. For example, the label 63 may be set so as to indicate the quality of the product R by a binary value. Further, for example, the label 63 may be set so as to indicate the probability that the product R is a non-defective product (that is, there is no defect) or the probability that the product R has a defect as a continuous value. Further, for example, the label 63 may be set to indicate a defective portion or the like.

The number of learning data sets 60 to be acquired may not be particularly limited and may be appropriately determined according to the embodiment. When the plurality of training data sets 60 are acquired, the control unit 11 proceeds to the next step S102.

(Step S102)
In step S102, the control unit 11 operates as the extraction unit 112, and extracts one or a plurality of first features 71 from the learning image 61 included in each learning data set 60.

FIG. 7 schematically illustrates an example of the feature extraction process according to the present embodiment. In the present embodiment, the control unit 11 extracts the first feature 71 from each learning image 61 by applying the convolution process to each learning image 61 as the feature extraction process. As a result, the one or more first features 71 are extracted as feature images having a size smaller than that of the learning image 61.

The convolution process corresponds to an arithmetic process for calculating the correlation between an image and a predetermined filter. For example, the control unit 11 may extract a feature image smaller in size than the learning image 61 from the learning image 61 by applying a convolution process to a part of the learning image 61. Alternatively, the control unit 11 extracts, for example, a feature image smaller in size than the learning image 61 from the learning image 61 by applying a convolution process to the learning image 61 and then further applying a pooling process to the obtained image. You may. The pooling process corresponds to an arithmetic process of aggregating a plurality of pixels in a target area into one pixel in order to obtain invariance of the response to a minute position change of a feature appearing in an image. The convolution processing and the pooling process may be executed by general image processing, or may be executed by using a learned learning model (for example, a convolutional neural network including a convolutional layer and a pooling layer). Further, the number of times each of the convolution treatment and the pooling treatment is applied may be appropriately selected according to the embodiment.

The vertical width (h) and the horizontal width (w) of the feature image are appropriately determined so as to be smaller than the vertical width (H) and the horizontal width (W) of the learning image 61, respectively. When the convolution process is applied to a part of the area of the training image 61, the size of the extracted feature image is determined according to the range to which the convolution process is applied. When the pooling process is applied, the size of the extracted feature image is determined according to the window size of the pooling process and the number of times the pooling process is applied. The control unit 11 may determine the size of the feature image to be extracted by appropriately determining these parameters of the convolution process and the pooling process. Each feature image may be referred to as a "feature map". The feature image corresponding to each first feature 71 may include an image relating to the defect.

The number of the first features 71 to be extracted may be appropriately set according to the embodiment. The control unit 11 can extract a plurality of different first features 71 from the learning image 61 by applying any of the above processes to the learning image 61 while changing each parameter. When a plurality of first features 71 are extracted from the learning image 61, the sizes of the feature images corresponding to each first feature 71 may be the same as each other or may be different from each other. After extracting one or a plurality of first features 71, the control unit 11 proceeds to the next step S103.

However, the type of the first feature 71 and the method for extracting the first feature 71 may not be limited to such an example. The type and data format of the first feature 71 is not particularly limited as long as it is some information that can be derived from the learning image 61 (particularly, information that can be related to an estimation target (defect in this embodiment)). It may be appropriately selected depending on the embodiment. Further, the method for extracting the first feature 71 may be appropriately selected according to the embodiment according to the type and data format of the first feature 71. For the extraction of the first feature 71, for example, general image processing (for example, edge extraction), SIFT, a trained learning model (for example, a convolutional neural network) or the like may be used.

(Step S103)
In step S103, the control unit 11 operates as the enlargement processing unit 113, and by applying the enlargement processing to the extracted one or a plurality of first features 71, one or a plurality of ones different from the one or a plurality of first features 71. Generate a plurality of second features 73.

The enlargement processing is information processing for deriving different features (second feature 73) from one or more features (first feature 71). For this enlargement processing, for example, a process of synthesizing a plurality of features, a process of converting features by geometric conversion, or the like may be used. Further, in the present embodiment, the first feature 71 is extracted as a feature image. Therefore, as the process of converting the feature, a process of synthesizing predetermined noise (white noise, random noise, etc.) into the feature image, a process of converting pixels included in the feature image, and the like may be used. Pixel conversion may include color conversion, density conversion, and the like, and a predetermined conversion function may be used for these conversions. In the present embodiment, the control unit 11 derives the second feature 73 from one or a plurality of first features 71 by adopting at least one of the following two methods.

(1) First Method First, the first method will be described with reference to FIG. 8A. FIG. 8A schematically illustrates an example of the enlargement processing by the first method (weighted synthesis). In the first method, weights are set for each first feature 71. The method for setting the weight for each first feature 71 does not have to be particularly limited, and may be appropriately selected depending on the embodiment. The weight for each first feature 71 may be randomly determined or may be determined by operator input. The control unit 11 weights each first feature 71 by applying the set weight to each first feature 71. Then, the control unit 11 generates a second feature 73 different from each first feature 71 by synthesizing each weighted first feature 71.

In the specific example of FIG. 8A, a scene in which three first features 711 to 713 (feature images) out of the plurality of first features 71 are synthesized is schematically shown. In this specific example, the control unit 11 multiplies each pixel of each first feature 711 to 713 (feature image) by each weight W1 to W3. Then, the control unit 11 synthesizes the weighted first features 711 to 713 (feature images) by adding the images obtained by this multiplication. As a result, the control unit 11 can generate a new second feature 73 (feature image) from each of the first features 711 to 713.

However, the number of the first feature 71 to be synthesized is not limited to three, and may be two or four or more. The control unit 11 may appropriately select a plurality of first features 71 to be used for generating the second features 73 from the plurality of first features 71 extracted in step S102. Further, the number of the second features 73 generated by the first method is not particularly limited and may be appropriately selected according to the embodiment. The control unit 11 can generate a plurality of different second features 73 by at least one of changing the weight applied to each first feature 71 and changing the first feature 71 to be combined. The first feature 71 to be synthesized may be randomly selected or may be selected by the operator's designation.

As shown in FIG. 8B, when the sizes of the feature images corresponding to the first feature 71 to be combined are different, the control unit 11 executes preprocessing for matching the sizes of the feature images. FIG. 8B schematically illustrates an example of a situation in which a plurality of first features 71 having different sizes are handled. In the specific example of FIG. 8B, scenes in which the sizes of the three first features 711 to 713 (feature images) are different are schematically shown. In this specific example, the control unit 11 adjusts the sizes of the first features 711 to 713 (feature images) so as to match each other by filtering the first features 711 to 713 (feature images). For the filtering for adjusting the size of the image, for example, filtering of known image processing such as nearest neighbor filtering may be used. The conversion process for adjusting the size of the image is not limited to the example using such filtering. For the conversion process for adjusting the size of the image, for example, a linear interpolation method such as bilinear interpolation or trilinear interpolation, or a network method such as Guided Backpropagation may be adopted.

The size of each of the first features 711 to 713 (feature images) after filtering may not be particularly limited and may be appropriately determined according to the embodiment. The size of each first feature 711 to 713 (feature image) after filtering may match any one of the first features 711 to 713 (feature image) before filtering, or the first feature before filtering may be the same. It may be different from any of 711 to 713 (feature image). As described above, when the sizes of the first features 71 to be combined are different, the control unit 11 executes a process of adjusting the sizes of the first features 71 as preprocessing. Then, the control unit 11 can generate the second feature 73 from each first feature 71 by applying the weighting synthesis process to each size-adjusted first feature 71.

(2) Second method Next, the second method will be described with reference to FIG. 9. FIG. 9 schematically illustrates an example of enlargement processing by the second method (geometric transformation). As shown in FIG. 9, in the second method, the control unit 11 applies a geometric transformation to one or more first features 71 to form one or more first features 71. Generate a different second feature 73. The geometric transformation is not particularly limited and may be appropriately determined according to the embodiment. For example, the geometric transformation may be translation, rotational movement, inversion, or a combination thereof.

The number of the second features 73 generated by the second method may not be particularly limited and may be appropriately selected depending on the embodiment. The control unit 11 can generate one second feature 73 by applying a geometric transformation to one first feature 71. The control unit 11 generates a plurality of different second features 73 by at least one of changing the first feature 71 to which the geometric transformation is applied and changing the parameters of the geometric transformation. be able to. The first feature 71 to which the geometric transformation is applied may be randomly selected or may be selected as specified by the operator. Further, the parameters of the geometric transformation define, for example, the amount of parallel movement, the amount of rotational movement, the presence or absence of inversion, and the like. The value of each parameter of the geometric transformation may be randomly determined or may be determined by operator input.

As described above, in the present embodiment, the control unit 11 adopts at least one of the first method and the second method, so that the control unit 11 has one or a plurality of first features different from the one or a plurality of first features 71. Two features 73 are generated from the one or a plurality of first features 71. When one or a plurality of second features 73 are generated, the control unit 11 proceeds to the next step S104.

(Step S104)
In step S104, the control unit 11 operates as the learning processing unit 114, and by performing machine learning, the corresponding label from at least one of one or a plurality of first features 71 and one or a plurality of second features 73. The estimator 50 trained to estimate the type of defect contained in the product R shown in the training image 61 is constructed so as to match 63.

First, the control unit 11 prepares an estimator 50 to be processed by machine learning. The configuration of the estimator 50 to be prepared, the initial value of the weight of the connection between each neuron, and the initial value of the threshold value of each neuron may be given by the template or by the input of the operator. Further, in the case of re-learning, the control unit 11 may prepare the estimator 50 based on the learning result data obtained by performing the past machine learning.

Next, the control unit 11 uses at least one of one or a plurality of first features 71 and one or a plurality of second features 73 derived from the training image 61 as input data (training data) for each training data set 60. It is used, and the corresponding label 63 is used as correct answer data (teacher data) to execute the learning process of the estimator 50. The number of features to be selected as input data, in other words, the number of features to be input to the estimator 50 may not be particularly limited and may be appropriately selected according to the embodiment. When a plurality of features are input to the estimator 50, the plurality of features may be composed of only the first feature 71, may be composed of only the second feature 73, or may be composed of the first feature 71 and the first feature. It may be composed of both of the two features 73. The number of features input to the estimator 50 corresponds to the number of features used to estimate the type of defect included in the product R shown in the learning image 61.

That is, the machine learning of the estimator 50 is performed by one or a plurality of features selected from one or a plurality of first features 71 and one or a plurality of second features 73 derived from the training image 61, and the corresponding label 63. A dataset is configured for use. Thereby, in the present embodiment, a plurality of data sets that can be used for machine learning of the estimator 50 can be derived from one learning data set 60. For example, in step S102, it is assumed that the control unit 11 has extracted four first features 71 from the learning image 61. Further, in step S103, it is assumed that the control unit 11 has generated 4 first features 71 to 6 second features 73. Further, it is assumed that the estimator 50 is configured to accept the input of one feature. In this case, the control unit 11 can derive 10 data sets from one learning data set 60. The control unit 11 executes the learning process of the estimator 50 by using such a data set derived from each learning data set 60. This stochastic gradient descent method or the like may be used.

For example, in the first step, the control unit 11 inputs the selected one or a plurality of features to the input layer 501 for each data set, and is included in each layer (501, 503, 505) in order from the input side. Judge the firing of neurons. As a result, the control unit 11 acquires an output value corresponding to the result of estimating the defect type from the selected one or a plurality of features from the output layer 505. In the second step, the control unit 11 calculates an error between the acquired output value and the value of the label 63. In the third step, the control unit 11 uses the error of the output value calculated by the error back propagation method to weight the connection between each neuron included in each layer (501, 503, 505) and the threshold value of each neuron. Calculate each error. In the fourth step, the control unit 11 updates the weight of the connection between each neuron and the value of each threshold value of each neuron according to each calculated error.

By repeating the first to fourth steps, the control unit 11 inputs one or a plurality of selected features for each data set, and outputs an output value that matches the corresponding label 63. Adjust the value of the parameter of the estimator 50. In other words, the control unit 11 determines the estimator 50 according to the first to fourth steps until the sum of the errors between the output value obtained from the output layer 505 and the value of the label 63 becomes equal to or less than the threshold value for each data set. Repeat the adjustment of the value of the parameter of. The threshold value may be appropriately set according to the embodiment. As a result, the control unit 11 is included in the product R shown in the training image 61 so as to match the corresponding label 63 from at least one of the one or a plurality of first features 71 and the one or a plurality of second features 73. An estimator 50 trained to estimate the type of defect can be constructed. When the machine learning of the estimator 50 is completed, the control unit 11 proceeds to the next step S105.

(Step S105)
In step S105, the control unit 11 operates as a storage processing unit 115, and stores information about the constructed learned estimator 50 in the storage unit 12 as learning result data 125. In the present embodiment, the control unit 11 generates information indicating the configuration and parameters of the estimator 50 constructed by machine learning in step S104 as learning result data 125. Then, the control unit 11 stores the generated learning result data 125 in the storage unit 12. As a result, the control unit 11 ends the process related to this operation example.

The storage destination of the learning result data 125 does not have to be limited to the storage unit 12. The control unit 11 may store the learning result data 125 in a data server such as NAS (Network Attached Storage), for example.

Further, after constructing the trained estimator 50, the control unit 11 may transfer the generated learning result data 125 to the inspection device 2 at an arbitrary timing. The inspection device 2 may acquire the learning result data 125 by receiving a transfer from the estimator generator 1, or may acquire the learning result data 125 by accessing the estimator generator 1 or the data server. good. The learning result data 125 may be incorporated in the inspection device 2 in advance.

Further, the control unit 11 may periodically update the learning result data 125 by periodically repeating the processes of steps S101 to S105. When this is repeated, the training data set 60 may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 11 may periodically update the learning result data 125 held by the inspection device 2 by transferring the updated learning result data 125 to the inspection device 2 every time the learning process is executed.

[Inspection equipment]
Next, an operation example of the inspection device 2 according to the present embodiment will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of the processing procedure of the inspection device 2 according to the present embodiment. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S201)
In step S201, the control unit 21 operates as the image acquisition unit 211 to acquire the target image 81 showing the product R to be the target of the visual inspection. In the present embodiment, the inspection device 2 is connected to the camera CA via the external interface 27. Therefore, the control unit 21 acquires the target image 81 from the camera CA. The target image 81 may be moving image data or still image data. When the target image 81 is acquired, the control unit 21 proceeds to the next step S202.

However, the route for acquiring the target image 81 does not have to be limited to such an example, and may be appropriately selected according to the embodiment. For example, another information processing device different from the inspection device 2 may be connected to the camera CA. In this case, the control unit 21 may acquire the target image 81 via another information processing device.

(Step S202)
In step S202, the control unit 21 operates as the extraction unit 212 and extracts one or a plurality of features 83 from the acquired target image 81.

This step S202 is processed in the same manner as in step S102. In the present embodiment, the control unit 21 extracts one or a plurality of features 83 from the target image 81 by applying the convolution process to the target image 81. Further, the number of features 83 to be extracted is determined so as to match the number of features selected as input data in step S104, that is, the number of features that the estimator 50 accepts input. After extracting one or a plurality of features 83, the control unit 21 proceeds to the next step S203.

(Step S203)
In step S203, the control unit 21 operates as the inspection unit 213, and the product R shown in the target image 81 contains a defect based on the extracted one or a plurality of features 83 using the trained estimator 50. Check if it is possible.

In the present embodiment, the control unit 21 sets the trained estimator 50 with reference to the learning result data 125. Next, the control unit 21 inputs the extracted one or a plurality of features 83 to the input layer 501 of the estimator 50, and determines the firing of each neuron included in each layer (501, 503, 505) in order from the input side. .. As a result, the control unit 21 acquires an output value corresponding to the result of estimating whether or not the product R shown in the target image 81 contains a defect (that is, whether the product R is good or bad) from the output layer 505 of the estimator 50. do. The control unit 21 inspects whether or not the product R shown in the target image 81 contains a defect based on the output value acquired from the output layer 505 of the estimator 50.

The quality of the product R may be inspected as appropriate according to the output format of the estimator 50. For example, when the output value obtained from the estimator 50 indicates the quality of the product R by two values, the control unit 21 can specify the quality of the product R according to the output value obtained from the estimator 50. Further, for example, when the output value obtained from the estimator 50 indicates the probability that the product R is a non-defective product or the probability that the product R has a defect as a continuous value, the control unit 21 has the output value obtained from the estimator 50. By comparing with the threshold value, the quality of the product R can be determined. Further, for example, when the output value obtained from the estimator 50 indicates a defective portion, the control unit 21 determines the quality of the product R shown in the target image 81 based on the output value obtained from the estimator 50. At the same time, if a defect exists, the location of the defect can be identified.

Further, for example, when the output value obtained from the estimator 50 indicates the index of the quality or defect type of the product R, the inspection device 2 determines the output value obtained from the estimator 50 and the quality or defect type of the product R. Reference information (not shown) such as a table format associated with may be stored in the storage unit 22. In this case, the control unit 21 can determine the quality of the product R according to the output value obtained from the estimator 50 by referring to this reference information.

As described above, the control unit 21 uses the trained estimator 50 to inspect whether or not the product R shown in the target image 81 contains a defect based on the extracted one or a plurality of features 83. Can be done. When the visual inspection of the product R is completed, the control unit 21 proceeds to the next step S204.

(Step S204)
In step S204, the control unit 21 operates as the output unit 214, and outputs information regarding the result of inspecting the quality of the product R in step S203.

The output format as a result of inspecting the quality of the product R does not have to be particularly limited, and may be appropriately selected depending on the embodiment. For example, the control unit 21 may output the result of inspecting the quality of the product R to the output device 25 as it is. Further, when it is determined in step S203 that the product R has a defect, the control unit 21 may issue a warning for notifying that the defect has been found as the output process of this step S204. Further, the control unit 21 may execute a predetermined control process according to the result of inspecting the quality of the product R as the output process of this step S204. As a specific example, when the inspection device 2 is connected to the production line for transporting the product, when the control unit 21 determines that the product R has a defect, the control unit 21 regards the defective product R as a defect-free product. May perform the process of transmitting the command to be conveyed by different routes to the production line as the output process of this step S204.

When the output processing of the information regarding the result of inspecting the quality of the product R is completed, the control unit 21 ends the processing related to this operation example. The control unit 21 may execute a series of processes of steps S201 to S204 each time the product R transported on the production line enters the shooting range of the camera CA. As a result, the inspection device 2 can inspect the appearance of the product R transported on the production line.

[feature]
As described above, in the present embodiment, the estimator 50 does not directly estimate the quality of the product R from the training image 61, but the first feature 71 and the first feature 71 and the first feature derived from the training image 61 in steps S102 and S103. 2 Trained to estimate the quality of product R from at least one of feature 73. In steps S102 and S103, the first feature 71 and the second feature 73 are derived so as to have a lower dimension than the training image 61 itself, so that the amount of information possessed by the first feature 71 and the second feature 73 is learned. It is less than the amount of information in the image 61. Therefore, as compared with the case where the training image 61 is trained to directly estimate the quality of the product R, in the present embodiment, the amount of information of the input data for the estimator 50 can be suppressed, so that the estimator can be suppressed. The configuration of 50 can be simplified. For example, the number of neurons contained in each layer (501, 503, 505) constituting the estimator 50 can be reduced.

In addition, in the present embodiment, by applying the enlargement processing to the first feature 71 extracted from the learning image 61 in step S103, the second feature 73 different from the first feature 71 is generated. As a result, it is possible to increase the variation of features for deriving the quality of the product R. That is, the number of training data used for machine learning in step S104 can be increased. Therefore, even if the number of learning data sets 60 (learning image 61) acquired in step S101 is small, it is possible to prepare a sufficient number of learning data to ensure a certain accuracy of defect estimation. Therefore, according to the estimator generator 1 according to the present embodiment, while reducing the calculation cost of the estimator 50, even if the number of learning images 61 is relatively small, the quality of the product R reflected in the image is highly accurate. It is possible to construct an estimator 50 that can be estimated. Further, the inspection device 2 according to the present embodiment can perform the appearance inspection of the product R in steps S201 to 204 with high accuracy and high speed by using the estimator 50.

§4 Modifications Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. For example, the following changes can be made. In the following, the same reference numerals will be used for the same components as those in the above embodiment, and the same points as in the above embodiment will be omitted as appropriate. The following modifications can be combined as appropriate.

<4.1>
In the above embodiment, the estimator 50 uses a so-called multi-layered fully connected neural network. However, the structure and type of the neural network constituting the estimator 50 are not limited to such an example, and may be appropriately selected depending on the embodiment. For example, the estimator 50 may utilize a convolutional neural network including a convolutional layer and a pooling layer.

Further, in the above embodiment, a neural network is used as a learning model constituting the estimator 50. However, the type of the learning model constituting the estimator 50 may not be particularly limited as long as machine learning for estimating the quality of the product R can be performed from each feature (71, 73). It may be appropriately selected according to the embodiment.

<4.2>
In the above embodiment, the learning result data 125 includes information indicating the configuration of the estimator 50. However, the configuration of the learning result data 125 does not have to be limited to such an example, and may be appropriately determined according to the embodiment. For example, when the configuration of the neural network to be used is common to each device, the learning result data 125 does not have to include information indicating the configuration of the neural network.

<4.3>
For each information processing (FIGS. 6 and 10) according to the above embodiment, steps can be omitted, replaced, and added as appropriate according to the embodiment. For example, the estimator generator 1 according to the above embodiment may directly acquire one or a plurality of first features 71 extracted from the training image 61 of each training data set 60. In this case, the extraction unit 112 may be omitted in the software configuration of the estimator generator 1. In addition, in the above information processing, the process of step S102 may be omitted.

FIG. 11 schematically illustrates an example of the software configuration of the estimator generator 1A according to this modification. The hardware configuration and software configuration of the estimator generator 1A according to the present modification may be the same as the estimator generator 1 according to the above embodiment, except that the extraction unit 112 is omitted. The estimator generator 1A according to the present modification is the same as the estimator generator 1 according to the embodiment, except that one or a plurality of first features 71 are directly acquired and the step S102 is omitted. May work with.

That is, in step S101, the control unit of the estimator generator 1A operates as the data acquisition unit 111, and the combination of one or a plurality of first features 71 and the label 63 extracted from the learning image of the product R, respectively. Acquire a plurality of configured training data sets 60A. In step S103, the control unit operates as the expansion processing unit 113, and by applying the expansion processing to one or a plurality of first features 71 included in each learning data set 60A, the control unit becomes one or a plurality of first features 71. Produces one or more different second features 73. In step S104, the control unit operates as the learning processing unit 114 and performs machine learning to perform machine learning from at least one of the one or a plurality of first features 71 and one or a plurality of second features 73, and the corresponding label 63. The estimator 50 trained to estimate the type of defect contained in the product R shown in the training image is constructed so as to be consistent with. In step S105, the control unit operates as the storage processing unit 115, and stores the information about the constructed trained estimator 50 as the learning result data 125. As a result, the estimator generator 1A according to the present modification can generate a trained estimator 50 that can be used for visual inspection of the product R, as in the above embodiment.

<4.4>
In the above embodiment, an example in which the present invention is applied to a scene where an appearance inspection of a product R shown in an image is performed is shown. However, the scope of application of the present invention is not limited to such an example of visual inspection. The present invention is applicable to all situations in which some attribute of an object in an image is estimated. By changing the image handled by the inspection system 100 from the image of the product R to the image of the object, an estimation system for estimating some attribute from the image of the object can be appropriately configured.

FIG. 12 schematically illustrates an example of an application scene of the estimation system 100B according to this modification. As shown in FIG. 12, the estimation system 100B according to the present modification includes an estimator generator 1B and an estimation device 2B connected via a network. The hardware configuration and software configuration of each device (1B, 2B) are the same as those of each device (1, 2) according to the above embodiment, except that the image handled is changed from the image of the product R to the image of the object. It may be the same. Further, each device (1B, 2B) may operate in the same manner as each device (1, 2) according to the above embodiment.

That is, the estimator generator 1B according to this modification acquires a plurality of learning data sets 60B each composed of a combination of a learning image 61B showing the object RB and a label 63B indicating the attributes of the object RB. .. Next, the estimator generator 1B extracts one or a plurality of first features 71B from the training image 61B included in each training data set 60B. The estimator generator 1B generates one or more second features 73B different from the one or more first features 71B by applying the enlargement processing to the extracted one or more first features 71B. Then, the estimator generator 1B learns from at least one of one or a plurality of first features 71B and one or a plurality of second features 73B so as to match the corresponding label 63B by performing machine learning. Construct an estimator 50B trained to estimate the attributes of the object RB shown in image 61B. The configuration of the estimator 50B may be the same as that of the estimator 50 according to the above embodiment.

Further, the estimator generator 1B may directly acquire one or a plurality of first features 71B as in the above <4.3>. That is, the estimator generator 1B is composed of a combination of one or a plurality of first features 71B extracted from the learning image 61B showing the object RB, and a label 63B indicating the attribute of the object RB, respectively. You may get a training data set. Next, the estimator generator 1B applies the enlargement processing to the one or a plurality of first features 71B included in each training data set, so that the one or a plurality of first features different from the one or a plurality of first features 71B are used. Two features 73B may be generated. Then, the estimator generator 1B learns from at least one of one or a plurality of first features 71B and one or a plurality of second features 73B so as to match the corresponding label 63B by performing machine learning. An estimator 50B trained to estimate the attributes of the object RB shown in image 61B may be constructed.

FIG. 13 schematically illustrates an example of the software configuration of the estimation device 2B according to this modification. The estimator generator 1B stores the information about the constructed estimator 50B as the learning result data 125B. The estimation device 2B according to this modification appropriately acquires the learning result data 125B. In step S201, the control unit of the estimation device 2B operates as the image acquisition unit 211 to acquire the target image 81B in which the object RB is copied. In step S202, the control unit operates as the extraction unit 212 and extracts one or a plurality of features 83B from the target image 81B.

In step S203, the control unit of the estimation device 2B operates as the estimation unit 213B, and the object RB reflected in the target image 81B is based on the extracted one or a plurality of features 83B using the trained estimator 50B. Estimate the attributes of. Specifically, the control unit sets the trained estimator 50B with reference to the learning result data 125B. Subsequently, the control unit inputs the extracted one or a plurality of features 83B to the trained estimator 50B, and executes the arithmetic processing of the trained estimator 50B. As a result of this arithmetic processing, the control unit of the estimation device 2B acquires an output value corresponding to the result of estimating the attribute of the object RB from the trained estimator 50B. In step S204, the control unit of the estimation device 2B outputs information regarding the result of estimating the attribute of the object RB. As a result, the estimation system 100B according to the present modification is configured to estimate some attributes from the image of the object RB.

The type of the object RB may not be particularly limited as long as it can be a target for estimating some attribute, and may be appropriately selected according to the embodiment. The object RB may be, for example, a product, a person, a body part of the person (for example, a face, etc.) to be subject to the visual inspection. Further, the "attribute" of the object to be estimated may be any property related to any property of the object, and may include all kinds of attributes of the object that can be estimated by the estimator. When the object is a person's face, the attributes to be judged are, for example, the type of facial expression, the position of facial parts (including organs) (including the relative positional relationship between specific organs), and facial parts. It may be the shape of the face, the color of the facial part, the state of the facial part (opening, angle, etc.), the attribute of the individual who owns the face, and the like. When the object is the body of a person, the attribute to be determined may be, for example, a pose of the body. When the object is a work to be worked on, the attributes to be determined may be, for example, the position and posture of the work.

100 ... Inspection system,
1 ... Estimator generator,
11 ... control unit, 12 ... storage unit, 13 ... communication interface,
14 ... input device, 15 ... output device, 16 ... drive,
111 ... Data acquisition unit, 112 ... Extraction unit,
113 ... Enlargement processing unit, 114 ... Learning processing unit,
115 ... Preservation processing unit,
121 ... Estimator generation program, 125 ... Learning result data,
2 ... Inspection equipment,
21 ... Control unit, 22 ... Storage unit, 23 ... Communication interface,
24 ... Input device, 25 ... Output device, 26 ... Drive,
27 ... External interface,
211 ... Image acquisition unit, 212 ... Extraction unit,
213 ... Inspection unit, 214 ... Output unit,
221 ... Inspection program,
50 ... Estimator,
501 ... Input layer, 503 ... Intermediate (hidden) layer, 505 ... Output layer,
60 ... Learning dataset,
61 ... learning image, 63 ... label,
71 ... 1st feature, 73 ... 2nd feature,
81 ... Target image, 83 ... Features,
91/92 ... Storage medium

Claims (8)

A data acquisition unit that acquires a plurality of learning data sets composed of a combination of a learning image showing a product to be visually inspected and a label indicating the type of defect contained in the product, and a data acquisition unit.
An extraction unit that extracts one or a plurality of first features from the training image included in each training data set.
An enlargement processing unit that generates one or more second features different from the one or more first features by applying the enlargement processing to the extracted one or more first features.
By performing machine learning, the product included in the product to be reflected in the training image so as to match the corresponding label from at least one of the one or more first features and the one or more second features. A learning processor that builds an estimator trained to estimate the type of defect,
Equipped with
Each of the one or a plurality of first features is extracted as a feature image having a size smaller than that of the learning image.
Weights are set for each of the plurality of first features.
Applying the enlargement process means matching the sizes of the plurality of first features, applying the set weights to each of the plurality of size-matched first features, and weighting the first feature. Including synthesizing multiple first features,
Estimator generator. Applying the enlargement process includes applying a geometric transformation to the one or more first features.
The estimator generator according to claim 1. The geometric transformation is translation, rotational movement, inversion, or a combination thereof.
The estimator generator according to claim 2. An image acquisition unit that acquires a target image of a product that is subject to visual inspection,
An extraction unit that extracts features from the acquired target image,
The estimator generator according to any one of claims 1 to 3, and the estimator generator.
Using the estimator constructed by the estimator generator, an inspection unit that inspects whether or not the product shown in the target image contains defects based on the extracted features, and an inspection unit.
An output unit that outputs information about the results of the inspection, and
To prepare
Inspection system . The computer
A step of acquiring a plurality of training data sets composed of a combination of a training image showing a product to be visually inspected and a label indicating the type of defect contained in the product, and a step of acquiring a plurality of training data sets.
A step of extracting one or more first features from the training image included in each training data set, and
A step of generating one or more second features different from the one or more first features by applying the expansion process to the extracted one or more first features.
By performing machine learning, the product included in the product to be reflected in the training image so as to match the corresponding label from at least one of the one or more first features and the one or more second features. Steps to build an estimator trained to estimate the type of defect,
And run
Each of the one or a plurality of first features is extracted as a feature image having a size smaller than that of the learning image.
Weights are set for each of the plurality of first features.
Applying the enlargement process means matching the sizes of the plurality of first features, applying the set weights to each of the plurality of size-matched first features, and weighting the first feature. Including synthesizing multiple first features,
Estimator generation method. On the computer
A step of acquiring a plurality of training data sets composed of a combination of a training image showing a product to be visually inspected and a label indicating the type of defect contained in the product, and a step of acquiring a plurality of training data sets.
A step of extracting one or more first features from the training image included in each training data set, and
A step of generating one or more second features different from the one or more first features by applying the expansion process to the extracted one or more first features.
By performing machine learning, the product included in the product to be reflected in the training image so as to match the corresponding label from at least one of the one or more first features and the one or more second features. Steps to build an estimator trained to estimate the type of defect,
To execute,
Each of the one or a plurality of first features is extracted as a feature image having a size smaller than that of the learning image.
Weights are set for each of the plurality of first features.
Applying the enlargement process means matching the sizes of the plurality of first features, applying the set weights to each of the plurality of size-matched first features, and weighting the first feature. Including synthesizing multiple first features,
Estimator generator. A data acquisition unit that acquires a plurality of learning data sets each composed of a combination of a learning image showing an object and a label indicating the attribute of the object, and a data acquisition unit.
An extraction unit that extracts one or a plurality of first features from the training image included in each training data set.
An enlargement processing unit that generates one or more second features different from the one or more first features by applying the enlargement processing to the extracted one or more first features.
By performing machine learning, the attributes of the object appearing in the training image can be determined from at least one of the one or more first features and the one or more second features so as to match the corresponding label. A learning processor that builds an estimator trained to estimate,
Equipped with
Each of the one or a plurality of first features is extracted as a feature image having a size smaller than that of the learning image.
Weights are set for each of the plurality of first features.
Applying the enlargement process means matching the sizes of the plurality of first features, applying the set weights to each of the plurality of size-matched first features, and weighting the first feature. Including synthesizing multiple first features,
Estimator generator. A data acquisition unit that acquires a plurality of learning data sets each composed of a combination of one or a plurality of first features extracted from a learning image of an object and labels indicating the attributes of the object, and a data acquisition unit.
An expansion processing unit that generates one or a plurality of second features different from the one or a plurality of first features by applying the enlargement processing to the one or a plurality of first features included in each learning data set. ,
By performing machine learning, the attributes of the object appearing in the training image can be determined from at least one of the one or more first features and the one or more second features so as to match the corresponding label. A learning processor that builds an estimator trained to estimate,
Equipped with
Each of the one or a plurality of first features is extracted as a feature image having a size smaller than that of the learning image.
Weights are set for each of the plurality of first features.
Applying the enlargement process means matching the sizes of the plurality of first features, applying the set weights to each of the plurality of size-matched first features, and weighting the first feature. Including synthesizing multiple first features,
Estimator generator.

Images