WO2021193347A1 - Data generation system, data generation method, data generation device, and additional learning requirement assessment device - Google Patents

Abstract

Provided is a method for creating learning data that can contribute to improving the recognition rate of a class assigned to an object. The present invention is provided with: a data extraction unit (130) that extracts first data relating to a bad local shape from data about the object having the bad local shape; and a data expansion unit (140) that combines the first data with second data about an object not having a bad local shape. The present invention is a data generation system (100) in which the data extraction unit (130) extracts the first data on the basis of the data about the object having the bad local shape, and position information data (170) about the bad local shape.

Description

Data generation system, data generation method, data generation device and additional learning necessity device

The present disclosure generally relates to a data generation system for learning data, a data generation method, a data generation device, and an additional learning necessity device having a data generation device.

In Patent Document 1, design data having three-dimensional information or mesh data having three-dimensional information imitating parts is rendered from many directions to generate a large number of images, and a learning target area is recognized in the generated images. A method of adding a label of a recognized learning target area to each image, further performing extended processing to obtain training data, modeling using the generated training data, and generating a classifier is disclosed.

Japanese Unexamined Patent Publication No. 2020-13467

Various sensors are used to determine the class assigned to an object. However, there is a problem that the judgment accuracy of the learner is lowered due to the difference in the sensor model, the noise removal process for the acquired sensor value, and the inspection environment. Further, in order to improve the determination accuracy of the learning device, learning data is required, but it is difficult to collect defective data for creating the learning data. Further, even if the AI (artificial intelligence) model is learned in advance, if the shape photographed in the customer environment is significantly different from the data used in advance, additional learning is required.

In order to solve the above problem, the data generation system of the present disclosure includes a data extraction unit and a data extension unit. The data extraction unit extracts the first data regarding the defective local shape from the data of the object having the defective local shape. The data extension unit combines the first data with the second data of an object having no defective local shape. The data extraction unit extracts the first data based on the data of the object having the defective local shape and the position information data of the defective local shape.

The learning data of the present disclosure has an advantage of improving the determination accuracy of the learning device that determines the class assigned to the object.

It is a schematic diagram of an example of the application target of the learning data according to the first embodiment of the present disclosure. It is explanatory drawing of an example of a defective product to be recognized. It is explanatory drawing of another example of a defective product to be recognized. It is explanatory drawing of still another example of a defective product to be recognized. It is a block diagram which shows the defective product data generation system which concerns on 1st Embodiment of this disclosure. It is a schematic diagram which shows an example of the image data contained in a teacher object in the same defective product data generation system. It is a schematic diagram which shows an example of the image data which imaged the good product of a bead in the same defective product data generation system. It is a schematic diagram which shows an example of the image data included in the defective product data generated by adding the additional image to the image data shown in FIG. 5A. FIG. 5 is a schematic diagram showing another example of image data included in defective product data generated by adding an additional image to the image data shown in FIG. 5A. It is a block diagram which shows the generation system of the non-defective product data which concerns on the 2nd Embodiment of this disclosure. It is a figure which shows the image in the vicinity of the bead of the good product sample which concerns on 3rd Embodiment of this disclosure. It is a figure which shows the enlarged image near the bead of the good product sample. It is a figure which shows the image in the vicinity of the bead of the defective sample which concerns on 3rd Embodiment of this disclosure. It is a figure which shows the enlarged image near the bead of the defective product sample. It is a figure which shows the actual image of the concave part in the vicinity of a bead of the defective product sample. It is a figure which shows the image of the virtual image corresponding to the concave part. It is a figure which shows the composite photograph of the enlarged image of the vicinity of the welding bead of a good product and the enlarged image of the vicinity of a welding bead of a defective product according to the same embodiment. It is a figure which shows the image of the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has a virtual image near the welded part in the welded sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has another virtual image in the vicinity of the weld part in the weld sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has another virtual image in the vicinity of the weld part in the weld sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the enlarged image of the region which has the protrusion of another shape in the vicinity of the weld part in the welding sample of another defective product which concerns on this embodiment. It is a figure which shows the image of the vicinity of the bead when the metal of two different materials which concerns on the same embodiment is welded with the metal of a different material. It is a figure which shows the enlarged image of the vicinity of the concave part of the bead when welding with the metal of a different material of the metal of two different materials which concerns on the same embodiment. It is a figure which shows the enlarged image of the virtual image obtained from the image of the concave part which the bead which concerns on this embodiment have. It is a figure which shows the enlarged image of the concave part when the virtual image is superposed on the image of the vicinity of the concave part of the bead which concerns on the same embodiment. It is a figure which shows the enlarged image of the concave part when the virtual image is superposed on the image of the vicinity of the concave part of the bead which concerns on the same embodiment. It is a figure which shows the enlarged image of the concave part when the virtual image is superposed on the image of the vicinity of the concave part of the bead which concerns on the same embodiment. The image obtained by cutting out the vicinity of the virtual image of the protrusion when the welded part is photographed into a rectangular shape is divided into M vertically and N horizontally at equal intervals and divided into M × N small square images for complementary processing. It is an explanatory diagram at the time of. It is explanatory drawing which shows the operation which cut out only the virtual image when the weld part was photographed. It is a figure which shows the image of the vicinity of the bead which concerns on a defective sample which concerns on this embodiment. It is a figure which shows the enlarged photograph of the vicinity of the concave part of the welding which concerns on the defective product sample which concerns on the same embodiment. It is the figure which cut out only the virtual image of the protrusion of the welding which concerns on the defective product sample which concerns on the same embodiment. It is a figure which shows the enlarged photograph of the recess D2 when the virtual image of the protrusion is superposed on the photograph of the vicinity of the recess of the welded portion which concerns on the same embodiment. It is a figure which shows the enlarged photograph of the recess D2 when the virtual image of the protrusion is superposed on the photograph of the vicinity of the recess of the welded portion which concerns on the same embodiment. It is a figure which shows the enlarged photograph of the recess D2 when the virtual image of the protrusion is superposed on the photograph of the vicinity of the recess of the welded portion. It is a block diagram which shows the additional learning necessity apparatus which concerns on 4th Embodiment of this disclosure, and the welding apparatus provided with the apparatus. It is a block diagram about the data generation part of the additional learning necessity apparatus which concerns on this embodiment. It is a block diagram concerning the data combination apparatus of the additional learning necessity apparatus which concerns on this embodiment. It is a figure which shows the flowchart about determining whether or not the additional learning necessity apparatus which concerns on this embodiment learns. It is a block diagram of the 1st means to obtain the defect local part data concerning this embodiment. It is a block diagram of the 2nd means which obtains the defect local part data concerning this embodiment. It is a block diagram which shows the additional learning necessity apparatus which concerns on 5th Embodiment of this disclosure, and the welding apparatus provided with the apparatus. It is a block diagram concerning the data generation part of the additional learning necessity apparatus which concerns on 5th Embodiment of this disclosure. It is a block diagram concerning the data combination apparatus of the additional learning necessity apparatus which concerns on 5th Embodiment of this disclosure. It is the schematic which shows the additional learning apparatus which concerns on 6th Embodiment of this disclosure, and the welding apparatus provided with the additional learning apparatus. It is the schematic which shows the additional learning apparatus which concerns on 7th Embodiment of this disclosure, and the welding apparatus provided with the additional learning apparatus.

Hereinafter, the learning data generation system according to the present disclosure will be described. It should be noted that all the embodiments disclosed below are examples, and there is no intention of limiting the learning data generation system according to the present disclosure.

Further, in the embodiment disclosed below, more detailed explanation than necessary may be omitted. For example, a detailed description of a well-known matter or a duplicate description of a substantially identical configuration may be omitted. This is to facilitate the understanding of those skilled in the art by avoiding unnecessarily redundant explanations.

(First Embodiment)
(1) Outline The learning data according to the first embodiment according to the present disclosure is the data that is the basis for determining the class assigned to the object. The "class" here is a classification such as a non-defective product and a defective product, and may be three or more classes such as a non-defective product, an adaptive product, and a defective product.

In the present embodiment, the target for determining the class using the learning data is, for example, as shown in FIG. 1, when two or more members (here, the first plate B11 and the second plate B12) are welded, the welded portion. The bead B1 formed in and the surrounding area. That is, FIG. 1 is a diagram showing an outline of the vicinity of the welded portion to which the learning data is applied. Then, when the image data including the bead B1 is input, the learner using the learning data estimates the state of the bead B1 and outputs the estimation result. Specifically, the learning device outputs, as an estimation result, whether the bead B1 is a non-defective product or a defective product, or if it is a defective product, the type of the defective product. That is, the learner is used for a weld appearance inspection to inspect whether the bead B1 is a non-defective product, in other words, whether or not welding is performed correctly.

Whether or not the bead B1 is a good product is determined by, for example, the length of the bead B1, the height of the bead B1, the rising angle of the bead B1, the throat thickness of the bead B1, the surplus of the bead B1, and the welding of the bead B1. It is determined by whether or not the numerical value such as the misalignment of the portion (including the misalignment of the start end of the bead B1) is within the permissible range. For example, if any one of the conditions listed above does not fall within the permissible range, the bead B1 is determined to be defective.

In addition to the judgment using the above numerical values, whether or not the bead B1 is a non-defective product is determined by, for example, whether or not the bead B1 has an undercut B2 (see FIG. 2A) and the pit B3 of the bead B1 (see FIG. 2B). It is also determined based on the presence or absence of spatter B4 (see FIG. 2C) of the bead B1 and the presence or absence of protrusions of the bead B1. For example, if any one of the defective parts listed above occurs, the bead B1 is determined to be a defective product.

Here, in order to generate a learning device with a high recognition rate, it is necessary to prepare a large number of image data including non-defective product data and defective product data to be recognized as learning data (see FIG. 3). However, when the frequency of defective products to be recognized is low, the learning data required to generate a learning device having a high recognition rate tends to be insufficient.

Therefore, it is conceivable to experimentally generate learning data imitating a defective product by CG (Computer Graphics) processing or data expansion processing. For example, it is conceivable to increase the number of training data and perform machine learning of the model by executing data expansion processing on the data obtained by actually photographing the bead B1 with the imaging device. The "data expansion process" here refers to a process of artificially inflating training data by adding processing such as translation, enlargement / reduction, rotation, inversion, or addition of noise to a certain data. ..

However, the problem is that the learning data obtained in this way is significantly different from the defective product data obtained in the actual inspection environment. The image data obtained from the sensors used in the inspection includes noise and virtual images. Therefore, depending on the model of the sensor used in the inspection and the noise removal processing method for the acquired image data, the learning device for class-determining the image to be inspected using the learning device generated using the learning data. The judgment accuracy will be different. In other words, in order to improve the judgment accuracy of the learner in the judgment of the class, it is necessary to generate the learning data in consideration of the difference in the inspection environment.

One of the embodiments according to the present disclosure is the defective product data generation system 100 in consideration of the difference in the inspection environment. FIG. 3 is a block diagram showing a defective product data 110 generation system 100 according to the first embodiment. In the present embodiment, a teacher object 120 having a locally defective shape (hereinafter, referred to as a defective local shape) imitating the bead B1 is generated in an environment different from the inspection environment. In the inspection environment, the inspector photographs the teacher object 120 using the sensors used in the inspection. The creator of the teacher object 120 extracts the defective local shape data 180 of the teacher object based on the data about the teacher object 120 obtained by photographing and the information about the position and range of the defective local shape in the teacher object. By combining the defective local shape data 180 and the non-defective product data 190 having no local shape, the defective product data 110 that contributes to the improvement of the determination accuracy of the learner is generated.

(2) Details Hereinafter, the defective product data generation system 100 will be described in detail with reference to FIG. The defective product data 110 is generated from the data extraction unit 130 and the data expansion unit 140.

The data extraction unit 130 includes the photographing unit 150, and generates the defective local shape data 180 based on the teacher object data 160 obtained from the photographing unit 150 and the position information data 170 of the defective local shape of the teacher object 120.

The teacher object 120 is an object determined by the learner (or, in other words, an object to be recognized). The teacher object 120 includes, for example, a bead B1 formed at a welded portion. When the bead B1 is used as a teacher object, the material of the teacher object 120 is metal, but in another embodiment, the teacher object 120 may be made of, for example, resin, and the material is not limited. Further, the teacher object 120 may be made of the same material, or may be made of a plurality of different materials. For example, a composite material of low carbon steel and high carbon steel may be used.

The teacher object 120 has a defective local shape. 2A to 2C are views showing an example of a cross section of the bead B1 having a defective local shape. As shown in FIGS. 2A, 2B and 2C, the defective local shape is, for example, undercut B2, pit B3, spatter B4, etc. generated during welding. However, the defective local shape is not limited to the above-mentioned shape, and may be, for example, a protrusion or melt-off. Further, a plurality of defective local shapes may be present in one teacher object 120, or a plurality of types of defective local shapes may be included.

The defective local shape may be created by imitating the defective local shape of the object to be inspected, or may be created by design. For example, when the object to be inspected is created by welding, the defective local shape of the teacher object 120 imitates the defective local shape of the defective product that occurs in the actual inspection environment by welding under the welding conditions where defects are likely to occur. Can be created.

The photographing unit 150 photographs the teacher object 120 and generates image information (teacher object data 160). At this time, the teacher object 120 having a defective local shape is provided to the inspection environment. In the inspection environment, the teacher object 120 is photographed using the imaging unit 150 used in the inspection. By this photographing, the teacher object data 160, which is the image information regarding the shape of the teacher object 120, is acquired. The image information may be three-dimensional image information or two-dimensional image information. Further, the teacher object data 160 may be three-dimensional point cloud data. The device used for shooting may be any device that can acquire image information, regardless of the type and performance of the device.

Although the case where the photographing unit 150 is an imaging device has been described here as an example, the present invention is not limited to this. For example, it may be configured to accept the input of data such as an image captured in advance by the imaging device used in the inspection. In this case, the photographing unit 150 can be referred to as a photographing data receiving unit.

It is preferable that the teacher object 120 is photographed under a plurality of imaging conditions. For example, the teacher object 120 is photographed at a plurality of photographing angles and a plurality of photographing positions. This makes it possible to efficiently collect information on the shape of one teacher object.

The creator or provider of the teacher object 120 owns the position information data 170 of the defective local shape of the teacher object 120, which is information regarding the position and range of the defective local shape of the teacher object 120.

The position information data 170 of the defective local shape is determined, for example, by the creator or provider of the teacher object 120. This will be described with reference to FIG. FIG. 4 is a schematic diagram showing an example of image data included in the teacher object 120 in the defective product data 110 generation system 100. The image data has a protrusion C1 as a defective local shape. The creator or provider of the teacher object 120 defines the protrusion C1 in FIG. 4 as a defective local shape, and stores information on the position and range of the defective local shape as position information data 170 of the defective local shape.

The data extraction unit 130 extracts the defective local shape data 180 from which the information related to the defective local shape is extracted from the teacher object data 160 based on the position information data 170 of the defective local shape. The defective local shape data 180 is image information unique to each inspection environment. In other words, even if the same teacher object 120 is used, the obtained defective local shape data 180 differs depending on the inspection environment. As described above, the defective local shape data 180 is image data related to the defective local shape, and can also be described as the first data.

The non-defective product data 190 is image data related to an object determined by the learner (or, in other words, an object to be recognized) without a defective local shape, and can also be described as a second data. The non-defective product data 190 is assumed to be data obtained in the inspection environment, but may be data obtained outside the inspection environment.

The data expansion unit 140 generates defective product data 110 by combining the above-mentioned defective local shape data 180 and non-defective product data 190.

This "combination" may be one in which the defective local shape data 180 and the non-defective product data 190 are added (combined), or the defective local shape data 180 is subjected to data expansion processing and then added (combined). It may be a thing. The data expansion process is performed by selectively using the parameters of the data expansion process.

In addition to the processing executed for the defective local shape data 180, the "data expansion processing" referred to in the present disclosure is newly learned data based on the parameters of the data expansion processing without using the defective local shape data 180. May include processing to generate. For example, the data expansion process may include a process of generating image data including a good bead B1 or an image data including a defective bead B1 by CG technology without using the first data. Further, in the data expansion process, for example, the data related to the defective shape may be expanded to the non-defective product data, or the data related to the shape other than the defective shape may be expanded to the non-defective product data.

Further, the "parameter of data expansion processing" referred to in the present disclosure is data expansion processing such as translation, enlargement / reduction, rotation, inversion, or addition of noise, which is executed for a part or all of the data to be processed. The degree of. For example, when the image data of the defective bead B1 having protrusions on the surface is used as the data to be processed, the parameters of the data expansion processing may include the amount of movement to move the protrusions, the size of the protrusions, the amount of rotation of the protrusions, and the like. .. Here, the parameters of the data expansion process are set in a range that can be changed for each type of process. For example, when the parameter is the movement amount for moving the protrusion, the movement amount can be changed in the range of 0 to several tens of mm. The parameter of the data expansion process may be one value.

In this embodiment, there may be a plurality of types of data expansion processing parameters. For example, in the data expansion process, in addition to the position and size on which the local shape generated based on the learning data is superimposed, one or more parameters are appropriately selected from the parameters such as aspect ratio, shape, angle, bottom shape, depth, and smoothness ratio. It can be selected and used. A range that can be changed is set for each of the plurality of types of parameters.

Using the first data, one or more parameters are selected from a plurality of types of parameters, and the data expansion processing is sequentially performed on the second data while changing the processing amount for each parameter within the changeable range. Will be executed. By this data expansion process, a large number of training data are generated based on one first data. According to the present embodiment, a large number of defective product data 110 are generated by expanding the defective local shape data 180 obtained from one teacher object 120 with respect to the non-defective product data 190.

As an example, it is assumed that a welded sample containing a defective local shape as shown in FIG. 4 is prepared. This welded sample is a defective product in which the protrusion C1 protrudes from the surface of the bead B1. The creator or provider of the weld sample possesses information about the location and extent of the protrusion C1 present in this weld sample. In the inspection environment, this weld sample is photographed and image information is acquired. Based on this image information and the information regarding the position and range of the protrusion C1, the image information regarding the shape of the protrusion C1 is extracted. Based on the information on the shape of the extracted protrusion C1, for example, by expanding the data on the image data of the welding sample having no defective local shape, the welding sample of the defective product having a new defective local shape is newly expanded. Image data is generated. At that time, by performing data expansion using one or a plurality of the parameters of the above-mentioned data expansion processing, a wide variety of defective product data 110 is generated from one teacher object 120. For example, it is possible to generate defective data 110 of a welding sample in which the size, position, angle and number of protrusions C1 are changed.

As an example, it is assumed that non-defective product data 190 including image data as shown in FIG. 5A exists. This image data is data of a so-called non-defective welded sample having no defective local shape. It is assumed that the data extraction unit 130 described above has extracted information regarding the shape of the protrusion C1 on the surface of the teacher object of the bead B1. By executing the data expansion process of adding the protrusion C1 to the non-defective bead B1 of FIG. 5A, it is possible to generate the image data as shown in FIG. 5B. At that time, the position, size, angle, and the like of the protrusion C1 can be freely changed as described in the above-mentioned description regarding the parameters of the data expansion process. The image data shown in FIG. 5B is stored in the data expansion unit 140 as defective product data 110.

Further, by creating the teacher object 120 with a plurality of materials, it is possible to generate defective product data 110 of an intermediate material. For example, when the teacher object 120 is made of low carbon steel and high carbon steel, it is possible to generate defective product data 110 made of carbon steel having an intermediate carbon content. Thereby, various types of defective product data 110 can be efficiently generated from one teacher object 120.

For example, as the defective local shape data 180 obtained from the teacher object 120, there are not only the protrusion C1 but also the pit D1 and the hole E1 as shown in FIG. 5C. It is also possible to select one or more protrusions C1, pits D1 and holes E1 and perform data expansion processing on the non-defective data 190 shown in FIG. 5A to obtain, for example, the image data shown in FIG. 5C. It is also possible to store the image data shown in FIG. 5C as defective product data 110 in the data expansion unit 140.

Further, by creating the teacher object 120 with a plurality of materials, it is possible to generate defective product data 110 of an intermediate material. For example, when the teacher object 120 is made of low carbon steel and high carbon steel, it is possible to generate defective product data 110 made of carbon steel having an intermediate carbon content. Thereby, various types of defective product data 110 can be efficiently generated from one teacher object 120.

(Second embodiment)
A block diagram showing the generation system 200 of the non-defective product data 210 according to the second embodiment of the present disclosure is shown in FIG.

In the second embodiment, it is possible to generate non-defective product data 210 that can further improve the recognition rate of the class by combining the data on the local shape of the non-defective product and the non-defective product data. In this case, as shown in FIG. 6, the first data corresponds to the non-defective local shape data 280, and the second data corresponds to the non-defective data 290. The position information data 270 of the defective local shape is data regarding the position and range of the defective local shape of the teacher object 220. Therefore, it can be said that the image information not extracted by the data extraction unit 130 is not the information regarding the defective local shape. In other words, in the process of extracting information on the defective local shape, information on shapes other than the defective local shape can also be acquired. The shape other than the defective local shape is a good local shape. In this case, the non-defective local shape data 280 is information regarding the shapes of dents and protrusions that are not recognized as defective and that exist in areas other than the position and range of the defective local shape. By performing data expansion processing on the non-defective product local shape data 280 with respect to the non-defective product data 290, the non-defective product data 210 is newly generated.

Therefore, in the second embodiment, the teacher object 220 is imaged by the photographing unit 250, features other than the defective local shape of the teacher object 220 are collected as the first learning data, that is, the teacher object data 260, and the good product data 290 is obtained. Data expansion processing can be performed. Since there are few non-defective products having no local shape in the actual welding process, the non-defective product data 210 generated in this embodiment has an effect of improving the determination accuracy of the learning device for non-defective products.

Note that the parameters of the data expansion process used in the generation of the defective product data 110 and the generation of the non-defective product data 210 are common, and it is possible to generate each data at the same time. That is, the learning data regarding defective products and non-defective products is efficiently generated from one teacher object.

(Third embodiment)
The defective product data generation system 100 and the data generation method according to the third embodiment of the present disclosure will be described. The defective product data generation system 100 according to the present embodiment is the same as the defective product data 110 generation system 100 according to the first embodiment shown in the block diagram of FIG. In the following, an example of a method of generating defective product data 110 will be mainly described.

(1) First Example The method of generating defective product data 110 in the first embodiment will be described below.

FIG. 7A shows an image (non-defective product data 190) in the vicinity of the bead B1 formed by welding the first plate B11 and the second plate B12 in a good product welding sample. Further, an image in which a part of the region R1 of the bead B1 is partially enlarged is shown in FIG. 7B. The first plate B11 and the second plate B12 are made of, for example, metal. The bead B1 is made of, for example, a metal.

On the other hand, FIG. 8A shows an image (teacher object data 160) of the vicinity of the bead B1 (teacher object 120) when the first plate B11 and the second plate B12 are welded in a defective welding sample having a recess D2. .. Further, the concave portion D2 in the bead B1 is imaged, and a partially enlarged image (defective local shape data 180) is shown in FIG. 8B. In FIG. 8B, a virtual image F2 is also shown.

Next, the virtual image F2 will be described with reference to FIGS. 9A and 9B. FIG. 9A shows an image showing the actual shape of the recess D2 of the bead B1 in the defective weld sample. FIG. 9B shows a photograph showing a virtual image F2 of the recess D2 of the bead B1 in a defective welded sample.

The actual defect of the second plate B12 is the recess D2 shown in FIG. 9A, but the recess D2 exists depending on the arrangement of the camera (shooting unit 150) included in the data extraction unit 130 of the generation system 100 and the light source inside the system. At the position, a virtual image F2 shown in FIG. 9B may be photographed instead of the actual appearance of the recess D2. Further, in the configuration in which the photographing unit 150 irradiates the object to be photographed with the laser and acquires the shape by the image formed by the irradiation or the time until the laser returns, the laser light repeatedly reflects or diffuses in the recess. A virtual image is created by doing so. Further, not only the concave portion D2 but also the virtual image F2 may be photographed due to a defect contained in the bead B1. This embodiment is an embodiment relating to the generation of defective product data 110 when the virtual image F2 is photographed.

By superimposing the image shown in FIG. 8B on the image shown in FIG. 7B, an image having the virtual image F2 shown in FIG. 10 can be obtained. That is, FIG. 10 is a diagram showing a composite image of the enlarged image of the weld related to the non-defective product in the vicinity of the bead B1 and the enlarged image of the weld related to the defective product in the vicinity of the bead B1. By superimposing this on the image data (non-defective product data 190) shown in FIG. 7A, image data indicating a defective product can be obtained. The data expansion unit 140 takes in this image data as defective product data 110 and learns it.

(2) Second Example The method of generating defective product data 110 in the second embodiment will be described below.

FIG. 11 shows a painter in the vicinity of the bead B1, which is the welded portion between the first plate B11 and the second plate B12, in another defective welded sample. In the welding sample shown in FIG. 11, various defects are present in the region surrounded by the square including the region R1A, the region R1B, and the region R1C. An enlarged image of region R1A is shown in FIG. 12A. An enlarged image of region R1B is shown in FIG. 12B. Further, an enlarged image of the region R1C is shown in FIG. 12C. 12A to 12C are diagrams showing an image of a virtual image F2 of a defect possessed by the bead B1.

Also, images of the virtual image F2 in another region are shown in FIGS. 13A, 13B, 13C and 13D. Images of the virtual image F2 in yet another region are shown in FIGS. 14A, 14B, 14C and 14D. By superimposing the image of the virtual image F2 in these regions with the image shown in FIG. 7B, an image having the virtual image F2 is obtained. By superimposing the image having the virtual image F2 on the image data shown in FIG. 7A, image data showing a defective product can be obtained. The data expansion unit 140 takes in this image data as defective product data 110 and learns it.

The images of FIGS. 12A to 12C, 13A to 13D, and 14A to 14D were cut out by identifying a plurality of regions from the image of one bead B1 shown in FIG. 11 taken by the photographing unit 150. It is an image. However, the shooting conditions of the shooting unit 150 may be variously changed to set a plurality of shooting conditions, and the bead B1 may be shot under different shooting conditions to obtain a plurality of images. For example, a plurality of positions of the photographing unit 150 may be set, and a predetermined area of the bead B1 may be photographed from each of the plurality of positions to obtain an image. The images thus obtained are similar to those shown in FIGS. 12A-14D.

(3) Third Example The method of generating defective product data 110 in the third embodiment will be described below.

In welding the first plate B11 and the second plate B12a and the second plate B12b made of two kinds of metals, a welding sample in which two kinds of metals are welded will be described.

FIG. 15A shows that the first plate B11 and the second plate B12a and the second plate B12b made of two different materials, specifically different metals, are made of a metal bead B1a and a metal different from the bead B1a. It is an enlarged view at the time of welding with the bead B1b. FIG. 15B is a diagram showing an image of a virtual image F2A corresponding to the recess D2A in the region R1D of the bead B1a. FIG. 15C is a diagram showing an image of a virtual image F2B corresponding to the recess D2B in the region R1E of the bead B1a.

16A to 16C are virtual images in which the virtual image F2A shown in FIG. 15B and the virtual image F2B shown in FIG. 15C are superimposed by multiplying them by a predetermined magnification. Specifically, it is an image of a virtual image when the height and width of the virtual image F2A are multiplied by α and the height and width of the virtual image F2B are multiplied by (1-α) (however, 0 <α <1). More specifically, FIG. 16A is an image of a virtual image when α = 0.2, FIG. 16B is an image of a virtual image when α = 0.5, and FIG. 16C is an image of a virtual image when α = 0.8. Is. The above is an example of a method of fusing the virtual image F2A and the virtual image F2B shown in FIG. 15C, and an intermediate image between the virtual image F2A and the virtual image F2B may be generated by using a method such as morphing.

By superimposing the images of these virtual images on the image data of non-defective products, image data showing a plurality of defective products can be obtained. The data expansion unit 140 captures and learns the image data indicating the plurality of defective products as defective product data 110.

(4) Fourth Example Next, a method of obtaining a virtual image and a method of generating welding data of defective products will be described with reference to FIGS. 17 to 20C as the fourth embodiment. The method of obtaining the virtual image and the method of generating welding data for defective products are performed by the data expansion unit 140.

FIG. 17 is a diagram showing an image in which the vicinity of the virtual image when the welded portion is photographed is cut out in a rectangular shape, and is divided into M vertically and N horizontally at equal intervals to form M × N small squares. It is explanatory drawing at the time of dividing into an image and performing a complementary process.

FIG. 18 is an explanatory diagram showing a process of extracting only the virtual image F2 from the real image.

(4-1) Complementary Value of Luminance First, as shown in FIG. 17, for a rectangular image including a virtual image, M × N images are arranged at equal intervals of M in the vertical direction and N in the horizontal direction. It is divided into small square images (hereinafter referred to as pixels). Note that M and N are integers. Next, the upper left pixel in FIG. 17 is labeled as (0,0), and the N pixels arranged in the horizontal direction are (1,0), ..., (n, 0), ... , (N-1,0) (n is an integer). Next, the leftmost pixel in FIG. 17, the second pixel from the top, is labeled as (0,1), and the pixels arranged in the horizontal direction are (1,1), ..., (N, 1). ), ..., (N-1,1). Similarly, next, the pixel on the far left of FIG. 17, which is m + 1th (m is an integer) from the top, is labeled as (0, m), and the pixels arranged in the horizontal direction are (1, m). ..., (n, m), ..., (N-1, m). Such labeling is performed from the top to the Mth, and M × N pixels arranged in a matrix are labeled. The pixels in the bottom row are (0, M-1), (1, M-1), ..., (N, M-1), ..., (N-1, M-). Labeled as 1). Hereinafter, the pixels arranged in a matrix of M × N are referred to as a pixel matrix.

Next, in the pixel matrix, the pixel at the outermost edge, that is,
Top line: (0,0), (1,0), ..., (n, 0), ..., (N-1,0)
Bottom line: (0, M-1), (1, M-1), ..., (n, M-1), ..., (N-1, M-1)
Leftmost column: (0,0), (0,1), ..., (0, m), ..., (0, M-1)
Rightmost column: (N-1,0), (N-1,1), ..., (N-1, m), ..., (N-1, M-1)
The brightness value is calculated as the complementary value P (n, m) for the pixels (1 ≦ m ≦ M-2, 1 ≦ n ≦ N-2) of the portion (n, m) excluding. The calculation method of the complementary value will be described below.

(4-2) Method of calculating complementary value First, the brightness of the pixels in the top row, bottom row, leftmost column, and rightmost column is obtained from the brightness of the actual image included in the pixel block. A real image is an actual image taken by a camera. The brightness is defined as, for example, 1 when the image is white and 0 when the image is black, and the target image is compared with the image obtained by the mixing ratio of white and black, and the mixture of white and black when they match. The value of the ratio. The brightness of the pixel (0, m) in the m + 1 row and the leftmost column obtained in this way is P (0, m), and the brightness of the pixel (N-1, m) in the m + 1 row and the rightmost column is determined. Let it be P (N-1, m). Further, the brightness of the pixel (n, 0) in the n + 1th column and the top row is P (n, 0), and the brightness of the pixel (n, M-1) in the n + 1 column and the bottom row is P (n, M-1). ). At that time, the complementary value P (n, m) of the brightness of the pixels (n, m) in the n + 1th row and the m + 1th column is

Given in. Obtaining P (n, m) in this way is called a method for calculating a complementary value. Further, in the pixel matrix, the brightness of the actual image is P (n, m) for the pixel at the outermost edge. The image including the complementary value P (n, m) obtained in this way is shown in FIG. 17 and FIG. 18 (b).

(4-3) Method for Obtaining Virtual Image of Protrusion Next, P (n, m) is subtracted from the brightness of the actual image ((a) in FIG. 18) relating to the pixel (n, m). When the subtracted value is used as an image, only a virtual image of the protrusion shown in FIG. 18C is obtained.

(4-4) Creation of Image Data of Defective Product FIG. 19A shows a photograph of the vicinity of the bead B1 when the first plate B11 and the second plate B12 are welded in a welding sample of a defective product.

In FIG. 19A, a photograph of the recess D2 in the region R1F of the bead B1 is shown in FIG. 19B. Further, the virtual image F2 obtained in (4-3) is shown in FIG. 19C.

The size of the virtual image F2 in FIG. 19C is variously changed, and it is superimposed on the photograph of the recess D2 shown in FIG. 19B. Then, images including virtual images F2 of various sizes shown in FIGS. 20A to 20C can be obtained.

By superimposing the images including the virtual image F2 shown in FIGS. 20A to 20C on the region R1F shown in FIG. 19A, welding image data showing a defective product can be obtained.

(4-5) Learning of welding image data indicating a defective product The data expansion unit 140 takes in and learns the welding image data indicating a defective product obtained in (4-4) as defective product data 110.

(Fourth Embodiment)
The learning data generation method according to the fourth embodiment of the present disclosure, the learning data generation system using the learning data generation system, and the additional learning necessity device including the learning data generation system will be described below. In addition, a welding device equipped with the additional learning necessity device will also be described.

FIG. 21 shows a block diagram of the additional learning necessity device 300 according to the fourth embodiment and the welding device 301 provided with the device 300. The welding device 301 is controlled by the controller 302. The additional learning necessity device 300 has a built-in defective product data generation system 100 shown in FIG. 3, and transmits data such as defective product data 110 to the controller 302. The defective product data 110 generation system 100 shown in FIG. 3 is shown in the first embodiment. The controller 302 receives data such as defective product data 110, and controls the welding device 301 based on the data.

Next, the additional learning necessity device 300 will be specifically described. The additional learning necessity device 300 includes a data generation device 101. The data generation device 101 has a built-in generation system 100 shown in FIG. 3, and has a data extraction unit 130 and a data expansion unit 140. Further, the data generation device 101 includes a central processing unit (CPU), a memory, and a communication device.

The data extraction unit 130 is an arithmetic unit that extracts desired data, and has a photographing unit 150 and a data generation unit 161. The photographing unit 150 is, for example, a camera provided with an image sensor, and images the teacher object 120. Examples of the image pickup device include a charge junction device (CCD) and a CMOS sensor.

FIG. 22 shows a block diagram of the data generation unit 161. The data generation unit 161 has, for example, an arithmetic unit 163 equipped with a CPU and a memory 162 such as a hard disk drive or SSD, and data of the teacher object 120 imaged by the photographing unit 150 (teacher object data 160). Is stored in the memory 162. Further, the data generation unit 161 acquires the position information data 170 of the defective local shape from the first data storage unit 171. The first data storage unit 171 has a memory such as a hard disk drive or an SSD, and stores the position information data 170 of the defective local shape. The data generation unit 161 calculates by combining the teacher object data 160 stored in the memory 162 and the position information data 170 of the defective local shape, generates the defective local shape data 180, and transmits the defective local shape data 180 to the data expansion unit 140.

The data expansion unit 140 is an arithmetic unit that generates defective product data 110, and has a second data storage unit 181, a third data storage unit 191 and a data combination device 111. The second data storage unit 181 has a memory such as a hard disk drive or SSD, and stores the defective local shape data 180 received from the data extraction unit 130. The third data storage unit 191 has a memory such as a hard disk drive or SSD, and stores non-defective data 190.

FIG. 23 shows a block diagram of the data combination device 111. The data combination device 111 has, for example, a calculation device 112 equipped with a CPU and a data expansion memory 113 such as a hard disk drive or SSD, and calculates by combining defective local shape data 180 and non-defective data 190. Defective product data 110 is generated. Then, the data combination device 111 learns the defective product data 110 as learning data by storing the generated defective product data 110 in the data expansion memory 113, and transmits the defective product data 110 to the controller 302.

The controller 302 has a central processing unit, a memory, and a communication device. When the controller 302 receives the defective product data 110 from the data combination device 111, it controls the welding device 301 to obtain data such as new non-defective product data 190 and position information data 170 of the defective local shape. These data are transmitted to the first data storage unit 171 and the third data storage unit 191 again.

In this way, the data generation device 101 acquires data related to welding and newly generates defective product data 110.

Then, the controller 302 controls the welding device 301 based on these data, and causes the welding device 301 to perform welding. The additional learning necessity device 300 compares the data of the bead B1 obtained as a result of welding with the defective product data 110, determines whether the welding is good or bad, and determines whether or not to perform re-learning.

Next, it will be described using the flowchart shown in FIG. 24 that the additional learning necessity device 300 determines whether the welded product is good or bad using the defective product data 110 and determines whether or not to relearn. ..

The additional learning necessity device 300 obtains the data of the bead B1 which is the result of welding by the welding device 301 (step S1). Further, the additional learning necessity device 300 obtains defective product data 110 by using the data generation device 101 (step S2). Here, in step S2, the data generation device 101 generates defective product data 110 according to the AI model constructed in advance. Here, the "pre-built AI model" may be an AI model previously provided in the additional learning necessity device 300, or may be, for example, an AI model learned immediately before by the third embodiment. good. Further, the "AI model" is a model that generates defective product data 110, compares the bead B1 data described below with the obtained defective product data 110, and infers good or bad. When the "AI model constructed in advance" is an AI model previously provided in the additional learning necessity device 300, the defective product data 110 is provided in the additional learning necessity device 300 in advance.

Next, the additional learning necessity device 300 compares the data of the bead B1 obtained according to the AI model constructed in advance with the obtained defective product data 110, and infers good / bad (step S3). ..

Next, the additional learning necessity device 300 determines whether the inference of good / bad obtained in step S3 is good (Yes) or bad (No) (step S4). Based on the result of good (Yes) or bad (No), it is determined whether or not to further store and learn the defective product data 110.

If the result in step 4 is good (Yes), that is, if the good product and the defective product can be accurately selected, the additional learning necessity device 300 instructs the data generation device 101 to stop the generation of the defective product data 110 (learning). Cancellation, step S5).

On the other hand, if the result in step 4 is defective, the additional learning necessity device 300 instructs the data generation device 101 to learn (step S6), that is, to newly generate defective product data 110 (step S7). Then, the additional learning necessity device 300 updates the AI model or constructs a new AI model (step S8, “new AI model”).

Note that, in step S4, a small number of data may be created by data expansion by the method of the present embodiment, and a "bad" determination may be made when the accuracy of inference for the created data is lower than a predetermined value. At that time, the data may be expanded to generate a larger amount of teacher data, and additional learning (re-learning) may be performed (steps S6 to S8). As a predetermined value, for example, the non-defective product rate of the bead B1 obtained by welding, that is, the number of non-defective products of the bead B1 is divided by the total number of the bead B1 to obtain a percentage.

It may be. Here, the "total number of bead B1" is the number of bead B1 actually welded, and the "good number of bead B1" is the number of bead B1 actually welded that is visually good. .. In this way, it is possible to determine whether the welding is good or bad with the same accuracy as visually confirmed without actually visually confirming the welding after learning.

The additional learning necessity device 300 can be said to be a learning data generation system when viewed as a system.

Next, a method for generating defective local shape data 180 will be described.

There are the following means to obtain the defective local shape data 180.

(First means)
The first means are the means shown below. That is, the bead B1 is photographed by using the photographing unit 150, and an image, that is, teacher object data 160 is obtained. The teacher object data 160 and the position information data 170 of a plurality of defective local shapes acquired from the first data storage unit 171 are collated by the data generation unit 161 and a plurality of defective local data are extracted from the teacher object data 160. .. Then, these plurality of defective local data are fused by the data generation unit 161 to obtain defective local shape data 180.

This first means will be described with reference to FIG. That is, FIG. 25 is a block diagram of means for obtaining defective local shape data 180.

The bead B1 is photographed by the photographing unit 150, and the teacher object data 160 is obtained. The obtained teacher object data 160 is, for example, an image shown in FIG. Next, the data generation unit 161 collates the teacher object data 160 with the position information data 170 of a plurality of defective local shapes, and obtains defective local data αi (i is an integer from 1 to n). The defective local data αi is, for example, an image shown in any of FIGS. 12A to 12C, 13A to 13D, and 14A to 14D. Then, the data generation unit 161 fuses the defective local data α1, α2, ..., αn to obtain the defective local shape data 180.

(Second means)
The second means is the means shown below. That is, a plurality of shooting conditions are set by variously changing the shooting conditions of the shooting unit 150 (for example, the position of the shooting unit 150, the emission intensity of the light source used for shooting, etc.), and the bead B1 is shot under different shooting conditions. To obtain a plurality of images, that is, a plurality of teacher object data 160. The teacher object data 160 may be an image of a part of the area of the teacher object 120. The plurality of teacher object data 160 and the position information data 170 of the defective local shape acquired from the first data storage unit 171 are collated by the data generation unit 161, and a plurality of defective local data from the plurality of teacher object data 160 are collated. Is extracted. Then, these plurality of defective local data are fused by the data generation unit 161 to obtain defective local shape data 180.

This second means will be described with reference to FIG. That is, FIG. 26 is a block diagram of means for obtaining defective local shape data 180.

In the photographing unit 150, it is assumed that there are n position information data 170 of the defective local shape, and the photographing conditions are set in n ways. Note that n is a natural number. The shooting condition is, for example, a shooting area. Let β1, β2, ..., Βn be the plurality of teacher object data 160 captured under these n conditions, respectively. The data βi (i is any integer from 1 to n) is, for example, an image shown in any of FIGS. 12A-12C, 13A-13D and 14A-14D. Next, the data generation unit 161 collates each data βi (i is an integer from 1 to n) with the position information data 170 of the defective local shape. Then, the data generation unit 161 fuses the data β1, β2, ..., δn to obtain defective local shape data 180.

It should be noted that in the first and second means described above, the case of n = 1 is also possible. At that time, the first means and the second means are the same.

Even if the additional learning necessity device 300 provided with the learning data generation method according to the above embodiment, the learning data generation system using the learning data generation system, and the learning data generation system has trained the Ai model in advance. This is especially useful when the shape taken in the customer environment is significantly different from the data used in advance.

(Fifth Embodiment)
The learning data generation method according to the fifth embodiment of the present disclosure, the learning data generation system using the learning data generation system, and the additional learning necessity device including the learning data generation system will be described below. In addition, a welding device equipped with the additional learning necessity device will also be described.

FIG. 27 shows a block diagram of the additional learning necessity device 400 according to the fifth embodiment and the welding device 401 provided with the device 400. The welding device 401 is controlled by the controller 402. The additional learning necessity device 400 has a built-in defective product data generation system 200 shown in FIG. 6, and transmits data such as non-defective product data 210 to the controller 402. The non-defective data 210 generation system 200 shown in FIG. 6 is shown in the second embodiment. The controller 402 receives data such as non-defective product data 210 and controls the welding device 401 based on the data.

Next, the additional learning necessity device 400 will be specifically described. The additional learning necessity device 400 includes a data generation device 201. The data generation device 201 has a built-in generation system 200 shown in FIG. 6, and has a data extraction unit 230 and a data expansion unit 240. Further, the data generation device 201 includes a CPU, a memory, and a communication device.

The data extraction unit 230 is an arithmetic unit that extracts desired data, and has a photographing unit 250 and a data generation unit 261. The photographing unit 250 is, for example, a camera provided with an image sensor, and images the teacher object 220. Examples of the image pickup device include a charge junction device (CCD) and a CMOS sensor.

FIG. 28 shows a block diagram of the data generation unit 261. The data generation unit 261 has, for example, an arithmetic unit 263 equipped with a CPU and a memory 262 such as a hard disk drive or SSD. The memory 262 stores the data (teacher object data 260) of the teacher object 220 imaged by the photographing unit 250. Further, the data generation unit 261 acquires the position information data 270 of the defective local shape from the first data storage unit 271. The first data storage unit 271 has, for example, a memory such as a hard disk drive or SSD, and stores position information data 270 of a defective local shape. The data generation unit 261 calculates by combining the teacher object data 260 stored in the memory 262 and the position information data 270 of the defective local shape, generates the non-defective local shape data 280, and transmits it to the data expansion unit 240.

The data expansion unit 240 is an arithmetic unit that generates non-defective data 210, and has a second data storage unit 281, a third data storage unit 291 and a data combination device 211. The second data storage unit 281 has a memory such as a hard disk drive or SSD, and stores non-defective local shape data 280 received from the data extraction unit 230. The third data storage unit 291 has a memory such as a hard disk drive or SSD, and stores non-defective data 290.

FIG. 29 shows a block diagram of the data combination device 211. The data combination device 211 has, for example, a calculation device 212 equipped with a CPU and a data expansion memory 213 such as a hard disk drive or SSD, and calculates by combining non-defective local shape data 280 and non-defective data 290. Generate non-defective product data 210. Then, the data combination device 211 learns the non-defective product data 210 as learning data by storing the generated non-defective product data 210 in the data expansion memory 213, and transmits the non-defective product data 210 to the controller 402.

The controller 402 has a central processing unit, a memory, and a communication device. When the controller 402 receives the non-defective product data 210 from the data combination device 211, it controls the welding device 401 to obtain data such as new non-defective product data 290 and position information data 270 of the defective local shape. These data are transmitted to the first data storage unit 271 and the third data storage unit 291 again.

In this way, the data generation device 201 acquires data related to welding and newly generates non-defective product data 210.

Then, the additional learning necessity device 400 causes the welding device 401 to perform welding based on these data. The additional learning necessity device 400 compares the bead B1 obtained as a result of welding with the non-defective product data 210, determines whether the welding is good or bad, and determines whether or not to relearn. The additional learning necessity device 400 uses the defective product data 110 to determine whether the welded product is good or bad, and determines whether or not to relearn, as in the fourth embodiment. That is, if the result of the good / bad determination is good, that is, if the good product and the defective product can be accurately selected, the additional learning necessity device 400 instructs the data generation device 201 to stop the generation of the good product data 210. If the result of the good / bad determination is bad, the additional learning necessity device 400 instructs the data generation device 201 to newly generate the good product data 210.

The additional learning necessity device 400 can be said to be a learning data generation system when viewed as a system.

The method of generating the defective local shape data 180 is the same as that of the fourth embodiment.

(Sixth Embodiment)
The additional learning necessity device 300 and the welding device 301 provided with the additional learning necessity device 300 according to the sixth embodiment of the present disclosure will be described.

FIG. 30 shows an outline of the additional learning necessity device 300 and the welding device 301 equipped with the device 300. In FIG. 30, the welding device 301 incorporates an additional learning necessity device 300. The additional learning necessity device 300 has a built-in controller 302. That is, the welding device 301 has a built-in additional learning necessity device 300 and a controller 302, which is different from the welding device 301 according to the fifth embodiment.

The functions of the additional learning necessity device 300 and the controller 302 are the same as those of the additional learning necessity device 300 and the controller 302 shown in the fifth embodiment.

(Seventh Embodiment)
The additional learning necessity device 300 and the molding device 501 provided with the additional learning necessity device 300 according to the seventh embodiment of the present disclosure will be described.

FIG. 31 shows an outline of the additional learning necessity device 300 and the molding device 501 equipped with the device 300. The molding device 501, for example, injection-molds a plastic product. That is, the molding apparatus 501 manufactures a product using a plastic raw material instead of the welding apparatus 301 welding using metal. That is, instead of welding, the additional learning necessity device 300 can be applied even in the case of manufacturing, for example, by injection molding a plastic product. Here, the position information data 170 of the defective local shape and the non-defective product data 190 can be obtained from the manufactured plastic product.

In the fourth, sixth and seventh embodiments, the additional learning necessity device 300 is applied to welding and injection molding of plastic products, but the additional learning necessity device 300 can be applied to other than that. It can be applied to the painting and adhesion of products made of various materials. Further, in the fifth embodiment, the additional learning necessity device 400 is applied to welding, but the additional learning necessity device 400 can be applied to other than that, for example, injection molding of plastic products and various materials. It can be applied to the painting and adhesion of various products.

Further, in the fourth to seventh embodiments, additional learning is required for the data such as the teacher object data 160, 260, the position information data 170, 270 of the defective local shape, the defective product data 110, and the non-defective product data 190, 290. It may be stored in the memory in the devices 300 and 400. Further, in the fourth to seventh embodiments, the teacher object data 160, 260, the position information data 170, 270 of the defective local shape, the defective product data 110, the non-defective product data 190, 290, etc. are stored in the external server. You may be. Further, in the fourth to seventh embodiments, the calculation and processing of the data may be performed by so-called cloud computing.

(Other embodiments)
The method of generating learning data according to the embodiment of the present disclosure has been described above based on the embodiment, but the present disclosure is not limited to these embodiments. As long as the gist of the present disclosure is not deviated, various modifications that can be conceived by those skilled in the art are applied to each embodiment, or a form constructed by combining components in different embodiments is also a form of one or a plurality of embodiments. It may be included in the range.

According to the present disclosure, when the inference result is not correctly obtained by the AI model constructed in advance for the data obtained by photographing the teacher object, the data is expanded by this method (superimposition) and additional learning (re-learning) is performed. Learning) can be done.

Further, according to the present disclosure, a data extension that creates a small number of data by the data extension according to the present embodiment and generates a larger number of teacher data when the accuracy of inference for the created data is lower than a predetermined value. Can be performed for additional learning (re-learning).

(Implementation mode)
The data generation system (100, 200) according to the first embodiment includes a data extraction unit (130) and a data expansion unit (140). The data extraction unit (130) extracts the first data (180, 280) regarding the defective local shape from the data (160, 260) of the objects (120, 220) having the defective local shape. Specifically, the data extraction unit (130) is based on the data (160, 260) of the objects (120, 220) having the defective local shape and the position information data (170, 270) of the defective local shape. 1 data (180, 280) is extracted. The data extension unit (140, 240) combines the first data (180, 280) with the second data (190, 290) of an object having no defective local shape.

In the data generation system (100, 200) according to the second embodiment, the object (120, 220) has a plurality of defective local shapes in the data generation system (100, 200) according to the first embodiment. Have.

The data generation system (100, 200) according to the third embodiment is the object (120, 220) in the data generation system (100, 200) according to the first embodiment or the second embodiment. , It is made of multiple materials.

In the data generation system (100, 200) according to the fourth embodiment, in the data generation system (100, 200) according to the first to third embodiments, the data extraction unit (130) is an object (120). , 220).

In the data generation system (100, 200) according to the fifth embodiment, in the data generation system (100, 200) according to the fourth embodiment, the photographing unit (150, 250) is an object (120, 220). ) Is shot under multiple shooting conditions.

The data generation system (100, 200) according to the sixth embodiment is the first data (180, 280) and the first data (180, 280) in the data generation system (100, 200) according to the first to fifth embodiments. The data (190, 290) of No. 2 is image information.

The data generation system (100, 200) according to the seventh embodiment is the first data (180, 280) and the first data (180, 280) in the data generation system (100, 200) according to the first to sixth embodiments. The combination of the two data (190, 290) is done by data expansion based on a plurality of parameters.

The data generation method according to the eighth aspect includes a step of extracting the first data (180, 280) regarding the defective local shape from the data (160, 260) of the objects (120, 220) having the defective local shape, and the first step. It has a step of combining the data of 1 (180, 280) and the second data (190, 290) of an object having no defective local shape. The extraction step is based on the data (160, 260) of the object (120, 220) having the defective local shape and the position information data (170, 270) of the defective local shape, and the first data (180, 280). Includes steps to extract.

In the data generation method according to the ninth aspect, in the data generation method according to the eighth aspect, the object (120, 220) has a plurality of defective local shapes.

In the data generation method according to the tenth aspect, in the data generation method according to the eighth or ninth aspect, the object (120, 220) is formed of a plurality of materials.

The data generation method according to the eleventh aspect is the data generation method according to the eighth to tenth aspects. Be prepared.

In the data generation method according to the twelfth aspect, in the data generation method according to the eleventh aspect, the photographing unit (150, 250) photographs the object (120, 220) under a plurality of photographing conditions.

The data generation method according to the thirteenth aspect is the data generation method according to the eighth to twelfth aspects, in which the first data (180, 280) and the second data (190, 290) are image information. ..

The data generation method according to the fourteenth aspect is the data generation method according to the eighth to thirteenth aspects, wherein the combination of the first data (180, 280) and the second data (190, 290) is plural. It is done by parameter-based data expansion.

The data generation method according to the fifteenth aspect is a method of generating learning data (110, 210) for determining a class assigned to a local feature of an object (120, 220). The method is based on a sample object (120, 220) having a class of local features and a sample object (120, 270) indicating the position and range of the object (120, 220) including the local features. , 220), and the learning data (110, 210) is generated using the local features (180, 280) obtained.

The data generation device (101) according to the sixteenth embodiment includes an extraction unit (130) and a data expansion unit (140). The extraction unit (130) extracts the first data (180) from the object (120) having a defective local shape. The extraction unit (130) includes a generation unit (161). The generation unit (161) extracts a plurality of data regarding the defective local shape from the object (120). The generation unit (161) fuses the plurality of data to generate the first data (180). The data expansion unit (140) includes a combination unit (111). The combination unit (111) combines the first data (180) and the second data (190) of an object having no defective local shape.

The data generation device (101) according to the seventeenth embodiment includes an extraction unit (130) and a data expansion unit (140). The extraction unit (130) extracts the first data (180) from the object (120) having a defective local shape. The extraction unit (130) includes a generation unit (161). The generation unit (161) generates the first data (180) by fusing the data (160) taken under a plurality of shooting conditions. The data expansion unit (140) includes a combination unit (111). The combination unit (111) combines the first data (180) and the second data (190) of an object having no defective local shape.

The additional learning necessity device according to the eighteenth embodiment infers the data (110) generated by the data generation device (101) according to the sixteenth or seventeenth embodiment using an AI model constructed in advance, and correctly infers the result. Is determined to perform additional learning when is not obtained.

The additional learning necessity device according to the nineteenth embodiment creates a small number of data in the additional learning device according to the eighteenth embodiment, and when the accuracy of inference for the created data is lower than a predetermined value, Perform additional learning by performing data expansion that generates more teacher data.

The additional learning necessity device according to the twentieth aspect includes the data generation device (110) according to the sixteenth or seventeenth, and the step (S1) of obtaining the data (B1) of the object and the first data (180). ), The data (B1) of the object and the first data (180) are compared, and the step (S3) of inferring good / bad is good or bad. Based on the step (S4) of determining whether or not the data is good or bad, it is determined whether or not to further store and learn the first data (180), and if the result is good, the additional learning is stopped (step). S5) and, if the result is poor, instruct the data generator (110) to newly generate the first data (S7) (S6), and update the AI model or build a new AI model. It has a step (S8) and.

The data generation system, data generation method, data generation device, and additional learning necessity device of the present disclosure can be applied to, for example, a welding appearance inspection for inspecting whether or not welding is performed correctly, and are industrially useful.

100, 200 Generation system 101, 201 Data generation device 110 Defective product data 111, 211 Data combination device 112, 212, 163, 263 Computing device 113, 213, 162, 262 Memory 120, 220 Teacher object 130, 230 Data extraction unit 140 , 240 Data expansion unit 150, 250 Imaging unit 160, 260 Teacher object data 170, 270 Poor local shape position information data 171, 271 First data storage unit 180 Defective local shape data 181, 281 Second data storage unit 190 , 210, 290 Good product data 191, 291 Third data storage unit 280 Good product local shape data 300, 400 Additional learning necessity device 301, 401 Welding device 302, 402 Controller 501 Molding device B1, B1a, B1b Bead B11 First plate B12, B12a, B12b 2nd plate B2 undercut B3 pit B4 spatter C1 protrusion D1 pit D2, D2A, D2B recess E1 hole F2, F2A, F2B imaginary image R1, R1A, R1B, R1C, R1D, R1E, R1

Claims (20)

1. A data extraction unit that extracts the first data related to the defective local shape from the data of the object having the defective local shape, and
   A data extension unit that combines the first data and the second data of an object having no defective local shape is provided.
   The data extraction unit
   The first data is extracted based on the data of the object having the defective local shape and the position information data of the defective local shape.
   Data generation system.
2. The object has a plurality of the defective local shapes.
   The data generation system according to claim 1.
3. The object is made of a plurality of materials,
   The data generation system according to claim 1 or 2.
4. The data extraction unit includes a photographing unit for photographing the object.
   The data generation system according to any one of claims 1 to 3.
5. The photographing unit photographs the object under a plurality of photographing conditions.
   The data generation system according to claim 4.
6. The first data and the second data are image information.
   The data generation system according to any one of claims 1 to 5.
7. The combination of the first data and the second data is performed by data expansion based on a plurality of parameters.
   The data generation system according to any one of claims 1 to 6.
8. A step of extracting the first data regarding the defective local shape from the data of the object having the defective local shape, and
   It has a step of combining the first data and the second data of the object having no defective local shape.
   The extraction step
   A step of extracting the first data based on the data of the object having the defective local shape and the position information data of the defective local shape is included.
   Data generation method.
9. The object has a plurality of the defective local shapes.
   The data generation method according to claim 8.
10. The object is made of a plurality of materials,
    The data generation method according to claim 8 or 9.
11. The data extraction unit includes a photographing unit for photographing the object.
    The data generation method according to any one of claims 8 to 10.
12. The photographing unit photographs the object under a plurality of photographing conditions.
    The data generation method according to claim 11.
13. The first data and the second data are image information.
    The data generation method according to any one of claims 8 to 12.
14. The combination of the first data and the second data is performed by data expansion based on a plurality of parameters.
    The data generation method according to any one of claims 8 to 13.
15. A method of generating training data to determine the class assigned to a local feature of an object.
    The learning data using the local features obtained by photographing the sample object by the sample object having the local features of the class and the position information indicating the position and range including the local features in the object. How to generate data.
16. An extraction unit that extracts the first data from an object with a defective local shape,
    The extraction unit has a data extension unit that combines the first data and the second data of an object that does not have the defective local shape, and the extraction unit extracts a plurality of data related to the defective local shape from the object. It includes a generator that fuses these multiple data to generate the first data.
    The data expansion unit includes a combination unit that combines the first data and the second data of an object having no defective local shape.
    Data generator.
17. An extraction unit that extracts the first data from an object with a defective local shape,
    The extraction unit has a data expansion unit that combines the first data and the second data of an object that does not have a defective local shape, and the extraction unit fuses data captured under a plurality of imaging conditions to obtain the first data. Including the generator that generates
    The data expansion unit includes a combination unit that combines the first data and the second data of an object having no defective local shape.
    Data generator.
18. An additional learning necessity device that makes an inference about the data generated by the data generation device according to claim 16 or 17, and determines that additional learning is performed when the inference result is not correctly obtained.
19. The additional learning according to claim 18, wherein a small amount of data is created, and when the accuracy of inference for the created data is lower than a predetermined value, data expansion is performed to generate a larger amount of teacher data, and additional learning is performed. Necessity device.
20. The step of providing the data generator according to claim 16 or 17 and obtaining the data of the object,
    The step of obtaining the first data and
    A step of comparing the data of the object with the first data and inferring good / bad, and
    The step of judging whether the inference of good or bad is good or bad, and
    Based on the good or bad result, it is determined whether or not to further store and learn the first data, and if the result is good, the additional learning is stopped.
    If the result is poor, the step of instructing the data generator to newly generate the first data, and updating the AI model or constructing a new AI model.
    Have,
    Additional learning device.

Images