KR20240093896A - Inspection method and information processing device for display device - Google Patents

Abstract

The display device inspection method is a display device inspection method that uses machine learning to determine which of j types of classification an image captured on the image display surface of the display device corresponds to, where the information processing device is A first machine learning step that performs machine learning to determine which of the j types of classification the image corresponds to, based on teacher data including image data corresponding to the type of classification, and an information processing device, and a second machine learning step that performs machine learning to determine which of the k types of classifications the image corresponds to, based on teacher data including data of images corresponding to k types of classifications, j and k. is a natural number greater than or equal to 2, and k is smaller than j.

Description

Inspection method and information processing device for display device

This disclosure relates to an inspection method and information processing device for a display device.

A method for evaluating luminance unevenness of a display device that displays an image is known (for example, patent document 1).

Japanese Patent Publication No. 2009-180583

Confirmation work to detect not only luminance unevenness but also display defects in the display device before shipment of the display device is performed as an inspection work of the display device. Additionally, there is a demand for automating display defects in display devices, such as the inspection work. In this case, automation of the task of classifying images captured on the image display surface of the display device is assumed. However, it is difficult to automatically classify images of non-defective products that are considered to have no display defects and images of non-defective products that are considered to have some display defects, especially from the viewpoint of ensuring classification accuracy.

The present disclosure has been made in view of the above problems, and aims to provide a display device inspection method and an information processing device capable of classifying images of the display device with higher precision.

A display device inspection method according to an aspect of the present disclosure is a display device inspection method that uses machine learning to determine which of j types of classification an image captured on the image display surface of the display device corresponds to, and includes information First machine learning in which a processing device performs machine learning to determine which of the j types of classifications an image corresponds to, based on teacher data including data of the images corresponding to the j types of classifications prepared in advance. A machine learning step, wherein the information processing device determines which of the k types of classification an image corresponds to, based on teacher data including data of the image corresponding to the k type of classification among the j types. It includes a second machine learning step that performs, where j and k are natural numbers of 2 or more, and k is smaller than j.

1 is a block diagram showing the main configuration of an information processing device.
Figure 2 is a block diagram showing the main functional configuration of the information processing device.
Fig. 3 is a diagram showing the classification of images in the decision processing by the determination unit and a schematic example of images of each classification.
Figure 4 is a graph showing the relationship between the grayscale value of the pixel before contrast enhancement and the grayscale value of the pixel after contrast enhancement.
Figure 5 is a block diagram showing an example of a more specific functional configuration of the determination unit.
Figure 6 is a schematic diagram showing the concept of hold-out verification.
Figure 7 is a schematic diagram showing the way of thinking of cross-validation.
Figure 8 is a table showing a way of thinking about the precision of judgment using machine learning.
Figure 9 is a flow chart showing the flow of machine learning to which multi-step processing is applied.
Figure 10 is a diagram showing the flow of machine learning to which multi-step processing is applied.
FIG. 11 is a diagram showing the correspondence between the actual class of verification data included in the reference data and the output (predicted class) of the first verification process by the process of step S3 explained with reference to FIG. 9.
FIG. 12 shows the actual class of verification data included in the “reference data corresponding to a specific output” input to the machine learning unit in the processing of step S6 explained with reference to FIG. 9, and the second processing by the processing of step S6. This is a diagram showing the correspondence between the output (predicted class) of the verification process (first time).
FIG. 13 shows the actual class of the verification data extracted from the process of step S4 explained with reference to FIG. 9 and input to the machine learning unit in the process of step S7, and the second verification process (second time) by the process of step S7. ) This is a diagram showing the correspondence between the output (predicted class).
FIG. 14 is a diagram showing the correspondence between the actual class of verification data included in the reference data and the final output (predicted class) obtained by the process of step S8 explained with reference to FIG. 9.
Fig. 15 is a flowchart showing the flow of decision processing by a machine learning unit in which multi-step processing is applied to classification judgment of image data using machine learning.
Figure 16 is a graph showing the relationship between the grayscale value of the pixel before contrast enhancement and the grayscale value of the pixel after contrast enhancement.
FIG. 17 is a diagram showing the relationship between the distribution of grayscale values in an image captured of a display panel with “uneven tilt” and the distribution of grayscale values in an image captured with a display panel of “defective product.”
Figure 18 is a diagram showing the difference between equation (1) and equation (2) and the relationship between the graph corresponding to equation (2) and gain_1, gain_2, Inflection_1, Inflection_2, height_1, and height_2.
FIG. 19 is a diagram showing an image captured of a display panel classified as a “defective product” and an example of application of image processing to the image.
FIG. 20 is a diagram showing an image captured of a display panel corresponding to both “scanning line defects” and “signal line defects” and an example of application of image processing to the image.
FIG. 21 is a diagram showing an image captured of a display panel classified as “inclined tilt,” and an example of application of image processing to the image.
FIG. 22 is a diagram showing an image captured of a display panel classified as “white unevenness” and an example of application of image processing to the image.
FIG. 23 is a diagram showing the relationship between a histogram showing the distribution of grayscale values of pixels included in an image captured from a display panel and the position (b) of an inflection point.

Below, each embodiment of the present disclosure will be described with reference to the drawings. In addition, the disclosure is only an example, and those skilled in the art that can easily imagine appropriate changes that maintain the main idea of the invention are naturally included in the scope of the present disclosure. In addition, in order to make the explanation clearer, the drawings may schematically display the width, thickness, shape, etc. of each part compared to the actual mode, but are only examples and do not limit the interpretation of the present disclosure. . In addition, in this specification and each drawing, elements similar to those described above with respect to previous drawings are given the same reference numerals, and detailed descriptions may be omitted as appropriate.

1 is a block diagram showing the main configuration of the information processing device 1. The information processing device 1 includes a communication unit 11, an input unit 12, a calculation unit 21, a storage unit 22, and an output unit 30. The information processing device 1 uses machine learning to capture images captured on the image display surface of a display device (e.g., image data included in judgment target image data 100 or reference data 223, which will be described later). It is an information processing device that can determine which of j types (e.g., j = 8) classification an image) corresponds to. Additionally, the information processing device 1 is an information processing device capable of performing contrast enhancement of the image.

The communication unit 11 communicates with external devices. Specifically, the communication unit 11 has a circuit that functions as, for example, a NIC (Network Interface Controller). The communication unit 11 outputs data received from an external device to the calculation unit 21. Additionally, when the information processing device 1 performs processing to transmit data to an external device, the communication unit 11 transmits the data to the external device. In addition, the external device may be, for example, a server different from the information processing device 1 or a stationary information processing device such as a PC (Personal Computer), or a portable terminal such as a smartphone, and may be a device not illustrated here. It may be an information processing device in another form.

The input unit 12 has an interface capable of inputting data. The interface is, for example, a bus interface that conforms to USB (Universal Serial Bus) or other standards, but is not limited to these and can be changed appropriately for specific standards. An external device is connected to the bus interface, for example, through a cable for inputting data from the external device. An input device, such as a keyboard or mouse, for an operator to operate the information processing device 1 may be further connected to the bus interface. At least one of the communication unit 11 and the input unit 12 functions as the acquisition unit 40.

The calculation unit 21 has a calculation circuit that functions as a CPU (Central Processing Unit). The arithmetic unit 21 reads the software/program and data referenced during execution processing of the software/program from the storage unit 22 and performs arithmetic processing corresponding to various functions exerted by the information processing device 1. Do. Hereinafter, when written as a program, etc., it refers to the software/program that the calculation unit 21 reads and executes, and the data referred to during execution processing of the software/program.

The storage unit 22 has a storage device that stores programs and the like. The storage device includes, for example, one or more of a solid state drive (SSD), a hard disk drive (HDD), and other rewritable storage devices, but programs, etc. are recorded thereon. It may also contain non-rewritable memory. In FIG. 1, a machine learning program 221, an image processing program 222, reference data 223, and setting data 224 are shown as programs stored in the storage unit 22. The arithmetic unit 21 and the storage unit 22 that stores programs etc. to be read by the arithmetic unit 21 function as the processing unit 50 . The machine learning program 221 is a program for performing machine learning. The image processing program 222 is a program for performing image processing on the judgment target image data 100, etc. Reference data 223 includes teacher data and validation data for machine learning. The setting data 224 includes various parameters related to execution processing of the machine learning program 221 and the image processing program 222.

The output unit 30 outputs output according to the processing contents of the processing unit 50. The output unit 30 is, for example, a monitor that displays images, but is not limited to this. The output unit 30 may further include a component that outputs in another form, such as a speaker that outputs audio, and when the communication unit 11 described above transmits data to an external device, the communication unit 11 It can be said that it functions as the output unit 30. Additionally, when the input unit 12 conforms to a standard that allows bidirectional data transmission, the input unit 12 also functions as the output unit 30.

FIG. 2 is a block diagram showing the main functional configuration of the information processing device 1. The information processing device 1 functions as an acquisition unit 40, a processing unit 50, and an output unit 30. The acquisition unit 40 acquires the judgment target image data 100. The communication unit 11 can acquire the judgment target image data 100 from an external device through communication. The input unit 12 can acquire judgment target image data 100 input from an external device.

The processing unit 50 functions as an image processing unit 51 and a determination unit 60. The image processing unit 51 performs image processing on the judgment target image data 100. The program for the calculation unit 21 to function as the image processing unit 51 is the image processing program 222. The determination unit 60 performs judgment processing to classify images using a neural network NN. The program for the calculation unit 21 to function as the determination unit 60 is the machine learning program 221. The image that is the target of judgment by the determination unit 60 is an image obtained by reading image data included in the judgment target image data 100.

The output unit 30 outputs the results of the decision processing by the determination unit 60. For example, the output unit 30 outputs a display indicating the result of the determination process by the determination unit 60.

FIG. 3 is a diagram showing the classification of images in the determination process by the determination unit 60 and a schematic example of an image for each classification. As shown in FIG. 3, the image is classified by judgment processing by the determination unit 60 into, for example, non-defective product, circuit defect, signal line defect, scan line defect, slant unevenness, vertical stripes, clusters, and white unevenness. do.

In the above-mentioned defect mode, tilt unevenness refers to a defect in which an area with a luminance difference with respect to the surroundings appears to extend along the long axis. Although it is written as “slant,” it also includes cases where it extends horizontally or vertically with respect to the screen. Vertical stripes refer to defects that are visible due to a difference in luminance occurring in response to divided blocks of a signal line, etc. A cluster refers to a defect that is identified by concentrating on an area with point defects and clustering them together. White non-uniformity refers to a defect that appears as a result of irregularly occurring areas that are brighter/darker than the surrounding area when uniform gradation display is performed within a plane. In the present invention, the detection of these types of unevenness is not limited, and defect modes not included in the above may be further defined.

The judgment target image data 100 includes a plurality of image data. The image data is image data obtained by capturing the image display surface of a display panel that is displaying a white or gray solid image. Here, the state of displaying a white solid image refers to the state in which all pixels are displaying and outputting at the highest gradation value of white. The state of displaying a half-tone solid image refers to a state in which all pixels are displaying and outputting with a gradation value that is less than the highest gradation value and exceeds the lowest gradation value. The display panel is, for example, an OLED (Organic Light Emitting Diode) display panel, but is not limited to this, and may be a display panel using another method. Each of the plurality of image data included in the judgment target image data 100 is an image of the image display surface of a different display panel. In the embodiment, the judgment processing performed by the judgment unit 60 is to confirm whether the plurality of display panels that are the target of imaging when generating the judgment target image data 100 can each perform display output with the assumed quality. This is a process intended to automatically perform inspection for the purpose. Hereinafter, when simply described as a display panel, it refers to one of a plurality of display panels that are the target of imaging when generating the judgment target image data 100. Additionally, when described simply as an image, unless otherwise noted, it refers to an image obtained by reading image data.

The judgment target image data 100 and the reference data 223 are either an image captured in the state of the mother glass substrate and cut out for each panel, or an image captured for each panel after the mother glass substrate was cut for each panel. You can continue.

The images of the image data included in the judgment target image data 100 and the reference data 223 are grayscale or black-and-white images. Additionally, when the image of the image data input to the information processing device 1 is a color image, the image processing unit 51, described later, may sequentially perform grayscale processing based on each primary color.

A good product refers to a product that can display and output with the expected quality. In other words, the display panel on which the image determined to be a good product is captured is a good product.

A circuit defect refers to a characteristic display defect occurring due to a defect in part or all of a driving circuit that supplies video signals or scanning signals to the pixels constituting the display screen. FIG. 3 shows a typical display example in which part or all of the image is a black area BA or a white area WA that is too bright for a white or halftone solid image display. A circuit defect occurs in the display panel where the image determined to be a circuit defect is captured.

A signal line defect refers to a characteristic display defect occurring due to a defect in a portion of a plurality of signal lines that transmit image signals to a plurality of pixels provided in the display panel. In Fig. 3, as the characteristic display defect, a black area BA that is significantly dark or a white area WA that is too bright for a white or midtone solid image display is generated in a straight line along the long side of a rectangular image. It shows an example. In addition, in the rightmost schematic example, a black area BA is generated along a direction perpendicular to the longitudinal direction.

Scanning line defects refer to characteristic display defects occurring due to defects in some of the plurality of scanning lines that transmit drive signals to the plurality of pixels provided in the display panel. In FIG. 3, as the characteristic display defect, a black area BA that is significantly dark in the display of a white or midtone solid image, or a dark area GA that is brighter than the black area BA but darker than the surroundings, is located along the long side of the rectangular image. A schematic example of generation occurring in a straight line along an orthogonal direction is shown.

As described above, the judgment target image data 100 is an image corresponding to a certain classification described with reference to FIG. 3 . The reference data 223 is prepared in advance on the assumption of classifying images of the image data included in the judgment target image data 100. Specifically, the reference data 223 is, for example, a display panel that corresponds to one of the classifications shown in FIG. 3 and is image data obtained by imaging the image display surface of a display panel whose classification has been confirmed in advance. Additionally, information that can specify the classification of the image data included in the reference data 223 when read by the calculation unit 21 is added.

The image processing unit 51 individually performs image processing on a plurality of image data included in the judgment target image data 100. Specifically, the image processing unit 51 performs processing to further emphasize the contrast of the image.

Figure 4 is a graph showing the relationship between the grayscale value of the pixel before contrast enhancement and the grayscale value of the pixel after contrast enhancement. When each gray scale value of a plurality of pixels included in an image before processing by the image processing unit 51 is displayed in 8 bits, as shown on the horizontal axis in FIG. 4, the gray scale value is in the range of 0 to 255. Take any of the values. If the contrast is not changed by the image processing by the image processing unit 51, the judgment target image data 100 is the gradation of the pixel before and after the contrast enhancement, as shown in the graph L4 shown in FIG. 4. There is no difference in price. The grayscale value after contrast enhancement is shown on the vertical axis in FIG. 4.

The image processing unit 51 changes each gradation value of a plurality of pixels included in the image so as to have a relationship before and after contrast enhancement that can be represented as, for example, any of the graphs L1, L2, and L3 shown in FIG. 4. By doing so, the contrast of the image is further emphasized. Graphs L1, L2, and L3 are sigmoid curves expressed by the sigmoid function shown in equation (1). y in equation (1) represents the grayscale value of the pixel after image processing by the image processing unit 51. x in equation (1) represents the grayscale value of the pixel before image processing by the image processing unit 51. b in equation (1) represents the value of x corresponding to the inflection point of the sigmoid curve, that is, x among the coordinates (x, y) at which the slope of the sigmoid curve is the largest. With b in between, the sigmoid curve is convex downward at x<b, and the sigmoid curve is convex upward at x>b. a in equation (1) is a coefficient corresponding to the magnitude of the slope of the sigmoid curve. In addition, exp[ ] in equation (1) and equation (2) described later indicates that it is an exponential function.

Additionally, in the description of the embodiment and equation (1) and equation (2) described later, it is assumed that the image of the image data included in the reference data 223 and the judgment target image data 100 is an 8-bit image. That is, the description of the embodiment and equations (1) and (2) assume that the highest grayscale value of the pixel included in the image is 255. The numerical value of “255” described as the numerator of equations (1) and (2) assumes the highest gray level value of the pixel included in the 8-bit image. However, images that can be handled by the information processing device 1 are not limited to 8-bit images. The information processing device 1 can handle r-bit images. r is any natural number. The highest gray level value of an r-bit image is ^2r -1. Therefore, when the image handled by the information processing device 1 is an r-bit image, the numerical value of “255” described as the numerator in equations (1) and (2) is replaced with “2 ^r -1” . In addition, the image of the image data included in the reference data 223 of the embodiment is previously subjected to image processing by the image processing unit 51.

FIG. 5 is a block diagram showing an example of a more specific functional configuration of the determination unit 60. The determination unit 60 includes a machine learning unit 70, a verification unit 61, and a limited classification processing unit 62. The machine learning unit 70 performs machine learning according to a predetermined algorithm. In the embodiment, CNN (Convolutional Neural Network) is adopted as the predetermined algorithm.

The machine learning unit 70 includes a feature extraction unit 71 and an identification unit 72. The processing included in CNN is roughly divided into two processes. One of the two processes is extraction processing of the feature quantity of the image that is the target of processing using CNN, and is performed by the feature quantity extracting unit 71. The other of the two processes is a process of obtaining the output of the neural network NN by receiving data representing the feature quantity of the image as input to the neural network NN, and is performed by the identification unit 72. In the embodiment, the output of the neural network NN is a probability value (within the range of 0 to 1) corresponding to each of the classifications described with reference to FIG. 3.

The feature extraction unit 71 includes a convolution processing unit 711 and a pooling processing unit 712. The convolution processing unit 711 performs convolution processing on the image. The convolution processing referred to here is the same as the convolution processing employed in general CNNs.

In convolution processing, each gray level value of a plurality of pixels included in an image is regarded as a matrix value. For example, assume a two-dimensional image in which p pixels are arranged in one (X direction) of two directions orthogonal to each other and q pixels are arranged in the other (Y direction). The two-dimensional image can be regarded as a matrix with the number of rows being p and the number of columns being q. Each gray level value of a plurality of pixels in the two-dimensional image can be regarded as a matrix component. If the matrix is α and the component of matrix α is β, the component at the e-th position in the X direction and the f-th position in the Y direction in matrix α can be expressed as β _ef . This indicates the gray level value of the pixel at the e-th position in the X direction and the f-th position in the Y direction in the two-dimensional image. Additionally, in the convolution process, a reference matrix called a kernel or filter is set as a different matrix from the two-dimensional image described above. Hereinafter, for convenience, the matrix for reference is referred to as the kernel. The number of rows of the kernel is smaller than the number of rows of the image that is the target of convolution processing (for example, the two-dimensional image described above). The number of columns of the kernel is smaller than the number of columns of the image that is the target of the convolution process (for example, the two-dimensional image described above). The elements of the matrix included in the kernel are values (generally integers) of a predetermined number of weights. The convolution processing unit 711 regards one of the images included in the judgment target image data 100 as a matrix, and performs extraction processing to extract a part of the matrix corresponding to the number of rows and columns of the kernel. In other words, this part can be regarded as a matrix with the same number of rows and columns as the kernel. The convolution processing unit 711 performs multiplication processing to obtain the product of the matrix obtained by multiplying the portion and the kernel. The convolution processing unit 711 performs addition processing to obtain one value by adding the values of the components included in the product of the matrix. The convolution processing unit 711 treats one value as one feature quantity. The convolution processing unit 711 sets a plurality of ranges to be subjected to extraction processing in one image regarded as a matrix. For example, in the case of p = q = 30, when the range of extraction processing is set in every other three cycles in the X and Y directions, the range of X × Y = 10 × 10 = 100 is set. The convolution processing unit 711 performs extraction processing, multiplication processing, and addition processing for each range to obtain the characteristic quantity of each range. In this way, one image is converted into data (first feature quantity matrix) in which values representing feature quantities are arranged in a matrix. Additionally, in more detail, padding (supplementation of pixels), etc. may be performed during the extraction process, but since this is well-known, detailed description is omitted here.

The pooling processing unit 712 performs pooling processing on the image on which the convolution processing has been performed by the convolution processing unit 711. The pooling process referred to here is the same as the pooling process employed in general CNNs.

In the pooling process, the components of the first feature matrix obtained by the above-described convolution process are divided into a plurality of groups. For example, assume that the first feature matrix is a matrix with 10 rows and 10 columns. Here, when the first feature matrix is divided into ranges of 2 rows and 2 columns, a range of 5×5 can be set for the first feature matrix. Here, each range is considered a separate group. In the pooling process, using this way of thinking, the components of the first feature matrix obtained by the convolution process can be divided into a plurality of groups. In the pooling process, one value is derived for each group. Examples of methods for deriving the single value include maximum value pooling, average value pooling, etc. Maximum value pooling is a method of deriving the maximum value among the elements of a matrix included in a group. Average value pooling is a method of deriving the average value of the elements of a matrix included in a group. By considering the values derived for each group as elements of a matrix, a matrix (second feature quantity matrix) with fewer rows and columns than the first feature quantity matrix can be derived through pooling processing. The second feature quantity matrix represents the feature quantity of one image (e.g., one of a plurality of images included in the judgment target image data 100) considered as a matrix by the convolution processing unit 711. It can be considered as data.

In addition, the processing of one image by the convolution processing unit 711 and the pooling processing unit 712 may be performed once or more, and may be repeated a predetermined number of times. Additionally, the processing performed by the feature quantity extraction unit 71 may further include the reverse operation of the convolution process (deconvolution) or the reverse operation of the pooling process (unpooling).

The identification unit 72 includes a neural network generation unit 721 and a probability conversion unit 722. The neural network generator 721 generates a neural network NN. The probability conversion unit 722 performs computational processing to convert the output of the neural network NN generated by the neural network generation unit 721 into a probability value (a value in the range of 0 to 1).

The input to the neural network NN when CNN is adopted is the component of the matrix (second feature matrix) obtained as the output of the above-described pooling process. The number of components included in the matrix corresponds to the number of input nodes of the neural network NN. A neural network NN is similar to a neural network in general machine learning and includes one or more fully coupled layers. In the pre-combined layer, weighting processing for the input is performed. In general, the number of biases of the fully connected layer (last stage) closest to the output side among one or more fully connected layers of a neural network NN is determined by the type of discrimination performed using the neural network NN (e.g., Figure 3 It is the same as the number of classifications described with reference to). In general, the output of a neural network NN can take any value (a number exceeding 1). Therefore, in order to treat the output of the neural network NN as “a value indicating certainty that can be used to determine which category it belongs to,” calculation by the probability conversion unit 722 is performed. The calculation by the probability conversion unit 722 is, for example, an calculation using a softmax function, but is not limited to this and can be changed appropriately.

The verification unit 61 provides a validation function to machine learning by the machine learning unit 70. The verification unit 61 suppresses overlearning in machine learning and makes it easier to increase the precision of judgment by machine learning. Examples of verification functions performed by the verification unit 61 include hold-out verification, cross-verification, and the like.

Figure 6 is a schematic diagram showing the concept of hold-out verification. In FIG. 6 and FIG. 7 described later, a data set DS including m pieces of image data is shown as a rectangle. In FIGS. 6 and 7, the first image data to the mth image data among the m pieces of image data are arranged in order along the long side direction of the rectangle, from one end side of the long side direction to the other end side. It is assumed that it is done. m is a natural number.

In hold-out verification, as shown in FIG. 6, the data set DS is divided into teacher data TD and verification data CD. Among these, teacher data TD is used as teacher data in machine learning. Verification data CD is used as verification data in machine learning.

One data set DS includes a plurality of image data corresponding to one decision result by machine learning. For example, the “good product” data set DS illustrated in FIG. 3 includes m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “good product.” Here, n of the m pieces of image data are extracted and used as teacher data TD, and the determination unit 60 performs machine learning using the teacher data TD. n is a natural number less than m. Then, among the data set DS, the determination unit 60 determines the verification data CD, which is (m-n) image data excluding the teacher data TD, after learning using the teacher data TD. Here, the verification data CD is a part of the data set DS and includes only image data corresponding to one decision result by machine learning. Therefore, ideally, the determination unit 60 after learning using the teacher data TD determines that all image data included in the verification data CD is image data corresponding to the one determination result. However, in reality, there are cases where part of the verification data CD is determined to be image data corresponding to a different determination result. Therefore, by determining the verification data CD in the decision unit 60 after machine learning using the teacher data TD, the decision accuracy of the decision unit 60 is verified, and the decision precision based on learning by the teacher data TD is further improved. What is done is done so that it can be done.

When hold-out verification is adopted, the verification unit 61 divides the data set DS into teacher data TD and verification data CD, and performs learning by the machine learning unit 70 using the teacher data TD and the verification data CD. The purpose is to verify the accuracy of the based judgment.

Figure 7 is a schematic diagram showing the way of thinking of cross-validation. When cross-validation is adopted, the verification unit 61 treats the data set DS as the data set DSS1, for example, as shown in FIG. 7. Data set DSS1 includes data sets DSa, DSb, DSc, DSd, and DSe. Data set DSa is data treated to divide data set DS into verification data CD1 and teacher data TD1. Data set DSb is data treated to divide data set DS into verification data CD2 and teacher data TD21 and TD22. The data set DSc is data that has been processed to divide the data set DS into verification data CD3 and teacher data TD31 and TD32. The data set DSd is data treated to divide the data set DS into verification data CD4 and teacher data TD41 and TD42. The data set DSe is data treated to divide the data set DS into verification data CD5 and teacher data TD5. Teacher data TD1, TD21, TD22, TD31, TD32, TD41, TD42, and TD5 are used as teacher data in machine learning. Verification data CD1, CD2, CD3, CD4, and CD5 are used as verification data in machine learning. Verification data CD1, verification data CD2, verification data CD3, verification data CD4, and verification data CD5 each include different image data. In other words, cross-validation sets a plurality of patterns (e.g., data sets DSa, DSb, DSc, DSd, DSe) to determine which part of the m image data included in the data set DS is considered verification data. And, the hold-out verification (see FIG. 6) is repeated as many times as the number of the plurality of patterns.

In the embodiment, the verification unit 61 that performs cross-verification is employed, but it is not limited to this, and the specific verification algorithm performed by the verification unit 61 can be changed as appropriate.

The reference data 223 of the embodiment includes a plurality of data sets corresponding to the number of classifications described with reference to FIG. 3 . Specifically, reference data 223 includes data sets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 (see FIG. 10). Data set DS1 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “defective products.” Data set DS2 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “slant unevenness.” Data set DS3 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “signal line defects.” Data set DS4 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “scanning line defects.” Data set DS5 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “circuit defects.” Data set DS6 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “vertical stripes.” Data set DS7 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to a “cluster.” Data set DS8 is a data set DS containing m image data obtained by individually imaging the image display surfaces of m display panels corresponding to “white unevenness.” The verification unit 61 in the embodiment individually cross-verifies each of the data sets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8.

Figure 8 is a table showing a way of thinking about the precision of judgment using machine learning. There is a condition where the case where an image “corresponds to an image in which a specific object was captured” is set to 1 (Positive), and the case where a certain image “does not correspond to an image in which a specific object was captured” is set to -1 (Negative). do. Under this condition, all images are considered to be images that can be given a decision of 1 or -1. In machine learning, an output is obtained that indicates a determination result of whether the image on which the input data (feature quantity) is based “corresponds to the image in which a specific object was captured.” The machine learning unit 70 produces output indicating a determination result of whether “a specific object corresponds to the captured image” according to the input. For a more specific example, the machine learning unit 70 of the embodiment determines the input data (feature quantity) according to the input of the judgment target image data 100 or verification data (for example, verification data CD, etc.). The image on which it is based falls under any of the following categories: “Good quality,” “Slope unevenness,” “Signal line defect,” “Scanning line defect,” “Circuit defect,” “Vertical stripes,” “Cluster,” or “White unevenness.” Produces output indicating recognition. Here, for example, an image determined to correspond to a classification other than a “good product” can be said to be an image determined to “do not correspond to a good product.”

The “Actual Class” column shown in FIG. 8 represents the results of a visual inspection by an inspector. For example, an image in which “a specific object was actually captured” is an image “corresponding to an image in which a specific object was captured” and is assigned 1. Additionally, an image in which “a specific object has not been imaged” is an image that “does not correspond to an image in which a specific object has been imaged” and is given -1. Meanwhile, “Predicted Class” shown in FIG. 8 represents the judgment result based on judgment using machine learning. In other words, 1 is given to an image determined in this judgment to “correspond to an image in which a specific object was captured.” In addition, -1 is given to an image for which it is determined that "the specific object does not correspond to the captured image" in this determination.

In fact, if the image in which “a specific object was captured” is determined to “correspond to an image in which a specific object was captured” in a decision using machine learning, the judgment is correct. In this case, when both the value given in the “actual class” and the value given in the “predicted class” are 1, it is called TP (True Positive). Additionally, if an image in which “a specific object is not captured” is determined to “do not correspond to an image in which a specific object is captured” in a decision using machine learning, the judgment is correct. In this case, when both the value given in the “actual class” and the value given in the “predicted class” are -1, it is called TN (True Negative). On the other hand, if the image in which “the specific object was captured” is determined to “do not correspond to the image in which the specific object was captured” in a decision using machine learning, the judgment is an error. In this case, when the value given in the “actual class” is 1 and the value given in the “predicted class” is -1, it is called FN (False Negative). Additionally, if an image in which “a specific object has not been imaged” is determined to “correspond to an image in which a specific object has been imaged” in a determination using machine learning, the determination is an error. In this case, when the value given in the “actual class” is -1 and the value given in the “predicted class” is 1, it is called FP (False Positive).

As concepts representing the precision of machine learning, for example, correct solution rate, recall rate, fit rate, false detection rate, F value, etc. are known. The value representing the correct solution rate is the value of (TP+TN)/(TP+TN+FN+FP). The value representing the recall rate is the value of TP/(TP+FN). The value representing the compliance rate is the value of TP/(TP+FP). The value representing the false detection rate is the value of FP/(TN+FP). If the detection rate is Q and the compliance rate is R, the F value is 2×Q×R/(Q+R).

The limited classification processing unit 62 performs multi-step processing. The multi-stage processing referred to here means extracting some input data corresponding to some of the decision results among the decision results shown by the output of the machine learning unit 70 for the plurality of input data from the plurality of images, and extracting the extracted This refers to a process of inputting some of the input data into the machine learning unit 70 and having the machine learning unit 70 further determine which of the partial determination results is correct.

For a specific example, the judgment target image data 100 includes a plurality of image data (for example, 1600 image data). First, the plurality of image data is input to the machine learning unit 70, so that an output indicating which of the classifications each of the plurality of image data has been determined to correspond to as described with reference to FIG. 3 is obtained. Here, when some image data (e.g., image data of 220) determined to correspond to a part of the classification described with reference to FIG. 3 (e.g., “good product” or “slant unevenness”) occurs, limited classification The processing unit 62 extracts the part of image data from the plurality of image data. The limited classification processing unit 62 inputs some of the extracted input data back into the machine learning unit 70 and causes the machine learning unit 70 to again output an output indicating a decision. However, in the judgment made according to the input of the part of the input data, the classification by the judgment is limited to a part of the classification (for example, “good product” or “slope unevenness”). In addition, if a part of the input data is simply input into the machine learning unit 70, the part of the input data is processed again by the convolution processing unit 711 and the pooling processing unit 712. Here, the data corresponding to this part of input data is adopted as is among the data after processing by the convolution processing unit 711 and the pooling processing unit 712 that have already been performed on the plurality of image data before extracting the part of input data. Thus, the processing by the convolution processing unit 711 and the pooling processing unit 712 may not be repeated.

Hereinafter, the flow of machine learning by the machine learning unit 70 to which the multi-step processing by the limited classification processing unit 62 is applied will be described with reference to FIGS. 9 and 10. The processing flow described with reference to FIGS. 9 and 10 is the processing flow of supervised learning using image data included in the reference data 223 as teacher data.

Figure 9 is a flow chart showing the flow of machine learning to which multi-step processing is applied. Figure 10 is a diagram showing the flow of machine learning to which multi-step processing is applied. As shown in Fig. 9, first, reference data 223 is acquired (step S1). Specifically, the calculation unit 21 that is executing the machine learning program 221 reads the reference data 223 from the storage unit 22. Here, as described above, some of the image data included in the reference data 223 are used as teacher data, and other parts are used as verification data. In FIG. 10, some of the data sets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 included in the reference data 223 are schematically shown as TD, and other data are treated as verification data. Some of them are schematically represented as CDs.

After the processing of step S1, the first-stage machine learning processing is performed by the machine learning unit 70 (step S2). The processing in step S2 is machine learning processing using teacher data included in the data sets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 included in the reference data 223. In the machine learning process, for example, when data representing the characteristic quantity of the image of the verification data included in the reference data 223 is input to the neural network NN, the image of the data is one of the eight types described with reference to FIG. 3. It is a machine learning process that determines the weighting value for the neurons of the neural network NN with the purpose of having the machine learning unit 70 determine which of the classifications. In Fig. 10, the first-stage machine learning process is described as “8-class CNN” and is denoted by step ST1.

After the processing of step S2, the first step verification processing is performed (step S3). In the processing of step S3, the calculation unit 21 transfers the verification data included in the data sets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 included in the reference data 223 to the machine learning unit 70. This is a process for obtaining an output from the machine learning unit 70 in which the machine learning through the process of step S2 is reflected as input. That is, the process of step S3 is a process of obtaining an output indicating which of the eight types of classifications the image of the verification data is determined to be by the machine learning unit 70 as described with reference to FIG. 3 . The output obtained by the processing in step S3 is shown as output OP1 in FIG. 10.

After processing in step S3, the limited classification processing unit 62 extracts verification data for which a specific output (verification result) was obtained in the first-stage verification process (step S4). The verification data extracted in the process of step S4 is a part of the verification data treated as input to the machine learning unit 70 in the process of step S3, and includes some of the eight types of classifications (e.g. For example, the output indicating that the product has been judged as “good product” or “slope unevenness” is the obtained verification data. In other words, “specific output” refers to output indicating what has been determined to be the particular classification.

After the processing in step S4, a second-stage machine learning processing is performed using image data corresponding to a specific output among the image data included in the reference data 223 (step S5). “Image data corresponding to a specific output among the image data included in the reference data 223” or “reference data corresponding to a specific output” is a part of the reference data 223, and is a part of the reference data 223, and in the process of step S4, the “image data corresponding to a specific output” refers to a data set (e.g., data set DS1 and data set DS2) containing image data corresponding to some classification (e.g., “good product” or “slant unevenness”) treated as “output of.” The processing in step S5 is machine learning processing using teacher data included in the “reference data corresponding to specific output” described above. In the machine learning process, for example, when data representing the characteristic quantity of the image of the verification data included in the reference data 223 is input to the neural network NN, the image of the data is one of the eight types described with reference to FIG. 3. Machine learning processing to determine the value of weighting in neurons of a neural network NN for the purpose of having the machine learning unit 70 determine which of the classifications corresponds to a “specific output”. am. In Fig. 10, the second-stage machine learning processing is described as “2-class CNN” and is designated as step ST2.

After the processing of step S5, the second-stage verification processing (first time) is performed (step S6). In the processing of step S6, the calculation unit 21 inputs verification data included in the above-described “reference data corresponding to a specific output” to the machine learning unit 70, and machine learning by the processing of step S5 is performed. This is a process to obtain the output of the reflected machine learning unit 70. That is, in the processing of step S6, the image of the verification data included in the above-described “reference data corresponding to a specific output” is classified by the machine learning unit 70 into eight types of classification explained with reference to FIG. 3, This is a process for obtaining an output that determines which of some classifications are considered to correspond to the “specific output” in the process of step S4.

Additionally, before the processing of step S7 described later, feedback to machine learning by the processing of step S5 based on the output obtained by the processing of step S6 may be performed. In other words, based on the processing in step S6, the accuracy of machine learning may be improved by the processing in step S5.

After the processing in step S6, a second-stage verification process (second time) is performed using the verification data extracted in the process in step S4 and from which a specific output (verification result) was obtained (step S7). In the processing of step S7, the calculation unit 21 inputs the verification data extracted in the processing of step S4 to the machine learning unit 70, and performs machine learning through the processing of step S5 and verification processing by the processing of step S6. This is a process to obtain the output of the machine learning unit 70 that has passed (the first time). That is, in the process of step S7, the image of the verification data extracted in the process of step S4, from which a specific output (verification result) is obtained, is classified by the machine learning unit 70 into the eight types explained with reference to FIG. 3. Among them, in the process of step S4, it is a process of obtaining an output of which of some classifications considered to correspond to the “specific output” is determined to be. The output obtained by the processing in step S7 is shown as output OP2 in FIG. 10.

After processing in step S7, the final output is the sum of the output obtained in the second-stage verification process (second time) and the output that is not a specific output (verification result) among the outputs obtained in the first-stage verification process. is performed (step S8). In the process of step S8, the calculation unit 21 outputs the output obtained by the process of step S7 and the output obtained by inputting the verification data that is not the target of extraction by the process of step S4 among the outputs obtained by the process of step S3. Integration is performed, and the integrated result is used as the final output of the machine learning unit 70. The output obtained by the processing in step S8 is shown as output OP3 in FIG. 10.

Additionally, when the verification function by the limited classification processing unit 62 is cross-validation, steps S2 to S7 are repeated depending on the number of data sets generated in cross-validation. The number of data sets can be, for example, for data set DSS1, data sets DSa,... , since the data set DSe is the target, it is 5.

Hereinafter, the change in precision of machine learning depending on whether or not multi-step processing is applied will be explained with reference to FIGS. 11 to 14. In the description with reference to FIGS. 11 to 14, data sets DS1, DS2,... , each data set DS of DS8 contains 1000 image data (m=1000), and among the 1000 image data, 800 are treated as teacher data and 200 are treated as verification data (number of judgment targets). is given as an example. In addition, the “number of correct answers” in the tables of Figs. 11 to 14 corresponds to TP in Fig. 8. In addition, “prediction precision” in the tables of FIGS. 11 to 14 is the ratio of the number of correct answers to the number of verification data (actual number of classes) corresponding to each classification, and corresponds to the recall rate described above.

FIG. 11 is a diagram showing the correspondence between the actual class of verification data included in the reference data 223 and the output (predicted class) of the first verification process by the process of step S3 explained with reference to FIG. 9. am. First, in the example shown in FIG. 11, FN (44/200) whose actual class is “good product” but whose predicted class is considered to be other than “good product” will be described. Among these FNs, there are nearly half of the FNs in which the predicted class was “slope non-uniformity” (21/44), which is significantly larger than the number of each of the other classes in the FNs. Next, in the example shown in FIG. 11, a TP in which the actual class and the predicted class match will be explained. The TP (111/200) of “slope unevenness” is significantly smaller than the TP of other categories. Next, in the example shown in FIG. 11, FN (89/200) whose actual class is “slope unevenness” but whose predicted class is considered to be other than “slope unevenness” will be explained. Among these FNs, the number of those whose predicted class was “good product” (62/89) exceeds half of the FNs, and is significantly larger than the number of each of the other classifications in this FN.

FIG. 12 shows the actual class of verification data included in the “reference data corresponding to a specific output” input to the machine learning unit 70 in the process of step S6 explained with reference to FIG. 9, and the process of step S6. This is a diagram showing the correspondence between the output (predicted class) of the second verification process (first time). As shown in the comparison of Fig. 11 and Fig. 12, by limiting the actual class and the predicted class to “good product” and “slope unevenness,” the degree of agreement between the actual class and the predicted class can be further increased. Specifically, in the example shown in FIG. 11, the fact that both the actual class and the predicted class are “good products” is 156/200, and when converted to prediction accuracy, this is 78.0%. On the other hand, in the example shown in FIG. 12, the fact that both the actual class and the predicted class are “good products” is 180/200, and when converted to prediction accuracy, this is 90.0%. Additionally, in the example shown in FIG. 11, the fact that both the actual class and the predicted class are “slope uneven” is 111/200, and when converted to prediction accuracy, this is 55.5%. On the other hand, in the example shown in FIG. 12, the fact that both the actual class and the predicted class are “slope uneven” is 160/200, and when converted to prediction accuracy, this is 80.0%.

FIG. 13 shows the actual class of verification data extracted from the process of step S4 explained with reference to FIG. 9 and input to the machine learning unit 70 in the process of step S7, and the second verification process by the process of step S7. This is a diagram showing the correspondence between the (second) output (predicted class). As shown in the comparison of Fig. 11 and Fig. 13, by limiting the actual class and the predicted class to “good product” and “slope unevenness,” the degree of agreement between the actual class and the predicted class can be further increased. Specifically, in the example shown in FIG. 11, the fact that both the actual class and the predicted class are “good products” is 156/200, and when converted to prediction accuracy, this is 78.0%. On the other hand, in the example shown in FIG. 13, the number of both the actual class and the predicted class being “good product” is 159/177, which when converted into prediction accuracy is 89.8%. Additionally, in the example shown in FIG. 11, the fact that both the actual class and the predicted class are “slope uneven” is 111/200, and when converted to prediction accuracy, this is 55.5%. On the other hand, in the example shown in FIG. 13, the fact that both the actual class and the predicted class are “good products” is 140/173, and when converted to prediction accuracy, this is 80.9%.

FIG. 14 is a diagram showing the correspondence between the actual class of verification data included in the reference data 223 and the final output (predicted class) obtained by the process of step S8 explained with reference to FIG. 9. The table shown in FIG. 14 is one in which some of the various values included in the table shown in FIG. 11 are updated based on the values included in the table shown in FIG. 13. By updating some of the values, among the values shown in FIG. 13, TP (159) of “good product” and FN of “good product”, that is, the actual class is “good product”, but the predicted class is “slope uneven.” ” (18), TP (140) of “slope unevenness,” and FN of “slope unevenness,” that is, the actual class is “slope unevenness,” but the predicted class is “good product” (33). It is reflected.

As shown in the comparison of Fig. 11 and Fig. 14, the final output obtained by the process of step S8 has higher prediction accuracy for “good product” and “slope unevenness” compared to the output of the first verification process. Therefore, even when looking at the overall prediction accuracy using machine learning, naturally, the final output obtained by the process in step S8 has a higher prediction accuracy compared to the output of the first verification process.

As explained with reference to FIGS. 9 to 14, by applying multi-step processing by the limited classification processing unit 62 in machine learning, prediction precision, that is, the degree of agreement between the actual class and the predicted class, can be further increased. You can. The multi-step processing by the limited classification processing unit 62 can be applied not only to machine learning but also to classification judgment of image data using machine learning. Hereinafter, the flow of decision processing by the machine learning unit 70 in which a multi-step process is applied to the classification decision of image data using machine learning will be described with reference to FIG. 15. Additionally, classification determination of image data using machine learning is performed after machine learning described with reference to FIGS. 9 to 14 . Additionally, in FIG. 10, the input flow of the judgment target image data 100 based on FIG. 15 is shown with a broken line.

Fig. 15 is a flowchart showing the flow of decision processing by the machine learning unit 70 in which a multi-step process is applied to the classification decision of image data using machine learning. First, the judgment target image data 100 is acquired (step S11). Specifically, the acquisition unit 40 acquires the judgment target image data 100 from an external device and outputs it to the calculation unit 21.

After the processing in step S11, image processing is performed by the image processing unit 51 (step S12). Specifically, as explained with reference to FIG. 4, the image processing unit 51 performs image processing to further emphasize the contrast of the image data included in the judgment target image data 100.

After the processing in step S12, the first stage of judgment processing is performed by the machine learning unit 70 (step S13). In the processing of step S13, the calculation unit 21 inputs the image data included in the judgment target image data 100 to the machine learning unit 70, and operates the machine learning unit ( 70) This is a process that obtains the output. That is, the processing of step S13 outputs which of the eight types of classifications the machine learning unit 70 determines that the image of the image data included in the judgment target image data 100 is of the eight types explained with reference to FIG. 3. This is the process of obtaining .

After the processing in step S13, the limited classification processing unit 62 extracts the judgment target image data 100 for which a specific output (judgment result) was obtained in the first-stage judgment processing (step S14). The verification data extracted in the process of step S14 is part or all of the image data included in the judgment target image data 100 treated as input to the machine learning unit 70 in the process of step S13, with reference to FIG. 3. Output showing that some of the eight types of classifications described (e.g., “good product” or “slant unevenness”) has been determined is the obtained image data. In other words, “specific output” refers to output indicating what has been determined to be the particular classification.

After the process in step S14, a second-stage determination process is performed using image data corresponding to the specific output extracted in the process in step S14 (step S15). In the process of step S15, the image of the image data extracted in the process of step S14 and for which a specific output (judgment result) has been obtained is classified by the machine learning unit 70 into the eight types described with reference to FIG. 3. This is a process for obtaining an output that determines which of some classifications are considered to correspond to the “specific output” in the process of step S14.

After the processing in step S15, processing is performed in which the final output is the sum of the output obtained in the second-stage judgment processing and the output that is not a specific output (verification result) among the outputs obtained in the first-stage judgment processing. Step S16). In the processing of step S16, the calculation unit 21 outputs the output obtained by inputting the output obtained by the processing of step S15 and the image data that is not the target of extraction by the processing of step S14 among the outputs obtained by the processing of step S13. Integration is performed, and the integrated result is used as the final output of the machine learning unit 70.

By going through the processing flow described with reference to FIG. 15 , compared to the case where images of image data included in the judgment target image data 100 are classified simply using machine learning, the case described with reference to FIGS. 11 to 14 Likewise, the precision of classification can be further increased.

In addition, in the description with reference to FIG. 4, graphs L1, L2, and L3 are exemplified as graphs showing the relationship between gray level values before and after contrast enhancement, but the relationship between gray level values before and after contrast enhancement is not limited to these.

Figure 16 is a graph showing the relationship between the grayscale value of the pixel before contrast enhancement and the grayscale value of the pixel after contrast enhancement. The image processing unit 51 changes the gray scale values of each of the plurality of pixels included in the image so as to have a relationship before and after contrast enhancement that can be represented as, for example, any of the graphs L11, L12, and L13 shown in FIG. 16. By doing so, the contrast of the image may be further emphasized.

In addition, the image processing performed by the image processing unit 51 in the processing of step S12, that is, contrast enhancement, is not limited to that corresponding to the above-described equation (1). Hereinafter, a more advanced way of thinking about contrast enhancement performed by the image processing unit 51 will be explained, but as a premise, the distribution of the gray scale value of the image captured for the display panel of “slant unevenness” and the display panel of “good product” The relationship between the distribution of grayscale values of the captured image will be explained with reference to FIG. 17.

FIG. 17 is a diagram showing the relationship between the distribution of grayscale values in an image captured of a display panel with “uneven tilt” and the distribution of grayscale values in an image captured with a display panel of “defective product.” The histogram Hs1 is a histogram showing the gray scale value of the pixel included in one image captured of the display panel with “slant unevenness.” The histograms referred to in the description of the embodiment, including the histogram Hs1, represent the range of gradation values of pixels included in the image (range from 0 to the maximum value that can be expressed by the number of bits of the gradation value) in the horizontal axis direction, The number of pixels for each gray level value is shown in the vertical axis direction. The histogram Hs2 is a histogram showing the grayscale values of pixels included in one image captured of a “good quality” display panel. The histogram Hs3 is a histogram showing the gray level values of pixels included in one image taken of a display panel of a “good quality” display panel and different from that of the histogram Hs2. Additionally, each histogram shown in FIG. 17 is a histogram showing the gray scale value of the image before image processing by the image processing unit 51 is performed.

As shown by the histograms Hs2 and Hs3 in FIG. 17, the distribution of gray scale values in an image taken of a “good product” display panel displaying a solid image in white or halftone is the histogram corresponding to the mode of occurrence of the gray scale values. Acid is produced significantly. The fluctuation range of the gray level value of the mountain of the histogram (the difference between the lowest gray level value and the highest gray level value of the mountain) is approximately equal to the number of bits of the pixel (the difference between the lowest gray level value and the highest gray level value that the pixel can take). In many cases, it is less than 10%. For example, if the number of bits of a pixel is 8 bits, the grayscale value takes a value of 0 to 255. On the other hand, there are few cases where the fluctuation range of the gray level value of the mountains of the histogram exceeds 25.

On the other hand, in an image captured of a display panel classified as “slope non-uniformity” such as histogram Hs1, the low gray level value range F11 is relatively low with respect to the gray level value corresponding to the mountain of the histogram corresponding to the mode of occurrence of the gray level value. The number of pixels with a gray level value occurs so significantly that it cannot be ignored. Additionally, in the histogram Hs1, there are few or no pixels included in the high gray-scale value range F12, which is relatively high with respect to the gray-scale value corresponding to the mountain of the histogram, on the side opposite to the low-gradation value range F11 across the mountain of the histogram, but there are few or no pixels in the surrounding area. In the case of a display panel in which tilt unevenness, which is visible at a relatively higher luminance, occurs, pixels included in the high grayscale value range F12 occur.

Additionally, there are few or no pixels included in the low gray scale value range F21 and the high gray scale value range F22 in the histogram Hs2, and the low gray scale value range F31 and the high gray scale value range F32 in the histogram Hs3. In other words, the difference between histogram Hs1 and histogram Hs2 is that in histogram Hs1, there are significantly pixels included in the low gray level value range F11, but in histogram Hs2, there are few or no pixels included in the low gray level value range F21. The “ends of the histogram,” such as the low-gradation value range F11, F21, and F31 and the high-gradation value range F12, F22, and F32, are within a range of gradation values within 30% of the number of pixel bits, for example, centered on the peak of the histogram. .

In Figure 17, for the purpose of showing that the low gray scale value range F11 and the low gray scale value range F21 are almost the same gray scale value range, there is a dashed line F1a connecting the lowest gray scale value of the low gray scale value band F11 and the lowest gray scale value of the low gray scale value band F21; A broken line F1b is shown connecting the highest gray level value of the low gray level value range F11 and the highest gray level value of the low gray level value range F21. In addition, for the purpose of showing that the high gray scale value range F12 and the high gray scale value range F22 are almost the same gray scale value range, a dashed line F2a connecting the lowest gray scale value of the high gray scale value range F12 and the lowest gray scale value of the high gray scale value range F22, and a high gray scale A dashed line F2b is shown connecting the highest gray level value of the value range F12 and the highest gray level value of the high gray level value range F22. The dashed lines F1a, F1b, F2a, and F2b almost follow the vertical axis of the graph.

Here, when comparing histogram Hs2 and histogram Hs3, the mountains of histogram Hs3 are overall shifted toward lower gradation values than those of histogram Hs2. In this way, even if the display panels are treated as “good products” in the same classification, there are individual differences in the gray scale values for each display panel. When calculating the gray scale value (y) after change based on equation (1) by contrast enhancement of a plurality of images captured from a plurality of display panels showing such individual differences, depending on the value of b, in the histogram Hs1 There are cases where an image occurs in which it is difficult to clearly distinguish between pixels included in the low gradation value range F11 and those without pixels included in the low gradation value range F31 in the histogram Hs3.

Therefore, the image processing unit 51 may perform contrast enhancement based on the following equation (2) instead of the above-mentioned equation (1). According to contrast enhancement based on equation (2), compared to equation (1), it becomes easier to more clearly distinguish whether pixels included in the above-described “end of the histogram” exist significantly. The difference between Equation (1) and Equation (2) and gain_1, gain_2, Inflection_1, Inflection_2, and height_1 in Equation (2) will be explained with reference to FIG. 18.

Figure 18 is a diagram showing the difference between equation (1) and equation (2) and the relationship between the graph corresponding to equation (2) and gain_1, gain_2, Inflection_1, Inflection_2, height_1, and height_2. Graph L21 shown in FIG. 18 is a graph showing the relationship between gray scale values before and after contrast enhancement based on equation (1), and is a graph corresponding to equation (1). Graph L21 is obtained by applying a=0.05 and b=128 to equation (1). Additionally, graph L22 is a graph showing the relationship between gray scale values before and after contrast enhancement based on equation (2), and is a graph corresponding to equation (2).

As shown in FIG. 18, graph L22 has two inflection points. One of the two inflection points is at the position of Inflection_1 with respect to the horizontal axis direction. The other of the two inflection points is at the position of Inflection_2 with respect to the horizontal axis direction. In other words, the values of Inflection_1 and Inflection_2 reflect the settings of the two inflection points in contrast enhancement based on equation (2). Here, Inflection_2 is larger than Inflection_1. gain_1 represents the size of the slope of one of the two inflection points. gain_2 represents the magnitude of the slope of the other of the two inflection points. gain_1 and gain_2 are the same values as a described above. height_1 represents the upper limit of the gradation value after contrast enhancement by one of the two inflection points and the lower limit of the gradation value after contrast enhancement by the other of the two inflection points. Additionally, the lower limit of the grayscale value after contrast enhancement by one of the two inflection points is 0. Additionally, the upper limit of the gray scale value after contrast enhancement by the other of the two inflection points is the highest gray scale value of the pixel. Inflection_1, Inflection_2, and height_1 take values that the grayscale value of the pixel can take, and are generally integers greater than 0.

In Figure 18, gain_1=0.5, gain_2=0.5, Inflection_1=90, Inflection_2=135, height_1=128. Additionally, in Figure 8, it is shown as height_2=(255-height_1)=127.

The histogram Hs4 shown in FIG. 18 is a histogram of an image captured of a display panel classified as a “good product.” As shown in FIG. 18, an image captured of a display panel classified as “good product” in a gradation value range with the gradation value corresponding to the value of Inflection_1 as the lower limit and the gradation value corresponding to the value of Inflection_2 as the upper limit. It is desirable that Inflection_1 and Inflection_2 are set so that the grayscale value range sufficiently includes the peaks of the histogram.

FIG. 19 is a diagram showing an image captured of a display panel classified as a “defective product” and an example of application of image processing to the image. The image before processing Be1 shown in FIG. 19 is an image before image processing by the image processing unit 51. The histogram Hs5 is a histogram showing the distribution of grayscale values of pixels included in the image Be1 before processing. The post-processed image Af11 is an image on which contrast enhancement has been performed by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be1 is the gradation value before contrast enhancement, and the relationship before and after contrast enhancement shown in the graph L21 is set. . The post-processed image Af12 is an image on which contrast enhancement has been performed by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be1 is the gradation value before contrast enhancement, and the relationship before and after contrast enhancement shown in the graph L31 is set. . Graph L31 is a graph corresponding to equation (2), gain_1=0.5, gain_2=0.5, Inflection_1=80, Inflection_2=140, height_1=128.

FIG. 20 is a diagram showing an image captured of a display panel corresponding to both “scanning line defects” and “signal line defects” and an example of application of image processing to the image. The image before processing Be2 shown in FIG. 20 is an image before image processing by the image processing unit 51. The histogram Hs6 is a histogram showing the distribution of grayscale values of pixels included in the image Be2 before processing. The post-processed image Af21 is an image on which contrast enhancement has been performed by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be2 is the gradation value before contrast enhancement, and the relationship before and after contrast enhancement shown by the graph L21 is set. . The post-processed image Af22 is an image on which contrast enhancement has been performed by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be2 is the gradation value before contrast enhancement, and the relationship before and after contrast enhancement shown in the graph L41 is set. . Graph L41 is a graph corresponding to equation (2), gain_1=0.5, gain_2=0.5, Inflection_1=90, Inflection_2=140, height_1=128.

The post-processed image Af21 shown in FIG. 20 becomes dark overall, making it more difficult to confirm the black area BA indicating scanning line defects compared to the pre-processed image Be2 before contrast enhancement. On the other hand, the post-processed image Af22 becomes brighter overall, making it easier to confirm the black area BA indicating scanning line defects. Additionally, the post-processed image Af22 is in a state in which it is possible to sufficiently confirm the white area WA indicating a signal line defect. Additionally, as shown in FIG. 19, there is no problem with equation (2) regarding confirmation of a “good product.” In this way, by performing contrast enhancement based on equation (2), it becomes easier to classify things other than "good quality" and "good quality" based on the light and dark of the image.

Additionally, when the value of height_1 is set to be equal to the highest gray level value of the pixel, gain_2 and Inflection_2 do not actually function. That is, in this case, equation (2) functions substantially the same as equation (1).

FIG. 21 is a diagram showing an image captured of a display panel classified as “inclined tilt,” and an example of application of image processing to the image. The image before processing Be3 shown in FIG. 21 is an image before image processing by the image processing unit 51. The histogram Hs7 is a histogram showing the distribution of grayscale values of pixels included in the image Be3 before processing. The post-processed image Af31 is an image that has been contrast-enhanced by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be3 is the gradation value before contrast enhancement, and has the relationship before and after contrast enhancement shown by the graph L21. . The post-processed image Af32 is an image that has been contrast-emphasized by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be3 is the gradation value before contrast enhancement, and has the relationship before and after contrast enhancement shown in the graph L51. . Graph L51 is a graph corresponding to equation (2), and gain_1=1, Inflection_1=100, and height_1=255.

FIG. 22 is a diagram showing an image captured of a display panel classified as “white unevenness” and an example of application of image processing to the image. The image before processing Be4 shown in FIG. 22 is an image before image processing by the image processing unit 51. The histogram Hs8 is a histogram showing the distribution of grayscale values of pixels included in the image Be4 before processing. The post-processed image Af41 is an image on which contrast enhancement has been performed by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be4 is the gradation value before contrast enhancement, and the relationship before and after contrast enhancement shown in the graph L21 is set. . The post-processed image Af42 is an image on which contrast enhancement has been performed by the image processing unit 51 so that the gradation value of the pixel included in the pre-processing image Be4 is the gradation value before contrast enhancement, and the relationship before and after contrast enhancement shown in the graph L61 is set. . Graph L61 is a graph corresponding to equation (2) and has gain_1=0.1, Inflection_1=105, and height_1=255. As shown in Figures 21 and 22, the unevenness can be made clearer by appropriately setting the position (Inflection_1) and the slope (gain_1) of the inflection point.

Hereinafter, the thinking method of how to determine the value of b in equation (1) and the value of Inflection_1 in equation (2) will be explained with reference to FIG. 23.

FIG. 23 is a diagram showing the relationship between a histogram Hs9 showing the distribution of grayscale values of pixels included in an image captured of the display panel and the position (b) of the inflection point. Additionally, histogram Hs9 is a histogram of an image captured of a display panel classified as “slant uneven.”

The range UN shown in FIG. 23 starts counting from the number of pixels with lower gradation values, and is the range of gradation values of pixels with less than 2000 pixels. Here, the width of the horizontal axis of the gray level value of the range UN with the lowest gray level value (0) of the pixel as the origin, that is, the highest value of the gray level value of the range UN is set as the value of b, and the histogram from which the range UN is derived (for example, For example, when equation (1) is adopted for contrast enhancement of an image from which the histogram Hs9) is derived, the inflection point of the graph can be set more appropriately. Additionally, when using equation (2) for contrast enhancement of an image, the highest value of the grayscale value in the range UN may be set as the value of Inflection_1. Additionally, by setting such a determination routine for b or Inflection_1 in the image processing unit 51, the inflection point in the contrast enhancement processing by the image processing unit 51 can be automatically set.

In addition, the fact that the number of pixels included in the range UN is less than 2000 is only an example and is not limited to this. The number of pixels included in the range UN is preferably determined as a ratio (for example, 15%) to the number of pixels in an image obtained by imaging the display panel.

Settings for various parameters, such as a, b in equation (1), gain_1, gain_2, Inflection_1, Inflection_2, height_1 in equation (2), and the number of pixels included in the range UN, are included in the setting data 224, for example. do. Rules related to image processing algorithms such as equation (1) and equation (2) are included in the image processing program 222, for example.

As described above, the method according to the embodiment uses machine learning to capture images captured on the image display surface of a display device (for example, images included in the judgment target image data 100 or reference data 223). and a display device inspection method for determining which of j types (e.g., j = 8) classification an image of data corresponds to. In the method according to the embodiment, the information processing device (e.g., information processing device 1) provides preliminarily prepared teacher data (e.g., reference data (e.g., reference data ( 223)), and includes a first machine learning step (for example, step S2) that performs machine learning to determine which of the j types of classification the image corresponds to. In the method according to the embodiment, the information processing device provides teacher data (e.g., reference data 223 ) and a second machine learning step (eg, step S5) that performs machine learning to determine which of the k types of classification the image corresponds to, based on the data sets DS1 and DS2). j and k are natural numbers of 2 or more. k is smaller than j. By these means, the classification accuracy of images can be further increased for classification of the k types among the j types. Therefore, images on the display device can be classified with higher precision.

Additionally, one of the k types (for example, k = 2) is a classification of display devices that are treated as good products that do not produce display output that is considered poor display. As a result, the classification accuracy of images can be further improved for k types of classification including non-defective products and one or more types of classifications other than non-defective products. Therefore, even for classifications in which it is relatively difficult to classify non-defective products among j types (e.g., j=8), such as the above-mentioned slope unevenness, the classification accuracy between non-defective products and non-defective products can be further improved.

In addition, the teacher data (e.g., data sets DS1, DS2 of the reference data 223) used in the second machine learning step (e.g., step S5) are used in the first machine learning step (e.g., step S5). It is part of the teacher data (e.g., reference data 223) used in S2). As a result, it is possible to commonize the teacher data used in the first machine learning step and the second machine learning step.

In addition, in the method according to the embodiment, the image captured of the image display surface of the display device (for example, the image of the verification data included in the reference data 223) is of j type (for example, j = 8). The information processing device (e.g., information processing device 1) determines which of the classifications it corresponds to after the first machine learning step (e.g., step S2) and also in the second machine learning step (e.g., step S2). , includes a first inspection step (eg, step S3) performed before step S5). In the method according to the embodiment, the information processing device determines k in the first inspection step among the data of the image (for example, verification data included in the reference data 223). It includes a first extraction step (e.g., step S4) for extracting data of an image determined to correspond to one of the classifications of types (e.g., k=2). The method according to the embodiment includes a second inspection step in which the information processing device determines which of the k types of classification the image data extracted in the first extraction step corresponds to after the second machine learning step. (e.g., step S7). Through these, it is possible to confirm the classification accuracy when an image classified as one of the k types in the first machine learning step is classified again after the second machine learning. Therefore, by reflecting the classification result of the second inspection step, the classification accuracy of the image can be further increased for the k types of classification compared to the case of only the first machine learning step.

In addition, the method according to the embodiment is data of an image captured on the image display surface of the display device, and data of an image different from the teaching data (e.g., reference data 223) (e.g., judgment target image data) A first decision step (e.g., information processing device 1) determines which of the j types (e.g., j = 8) classification (100)) corresponds to , step S13). The method according to the embodiment is to select an image determined to correspond to any of k types (for example, k = 2) of the classification in the first judgment step among the data of the image that was the judgment target in the first judgment step. It includes a second extraction step (eg, step S14) for extracting data. The method according to the embodiment includes a second determination step (for example, step S15) in which the information processing device determines which of the k types of classification the data of the image extracted in the second extraction step corresponds to. do. By these means, the classification accuracy of images can be further increased for classification of the k types among the j types. Therefore, images on the display device can be classified with higher precision.

In addition, the information processing device 1 of the embodiment uses machine learning to capture an image captured on the image display surface of a display device (for example, included in the judgment target image data 100 or reference data 223). It is an information processing device that can determine which of j types (e.g., j=8) classification an image of image data corresponds to. The information processing device 1 has a storage unit (for example, a storage unit (for example, a storage unit ( 22)) and a calculation unit (for example, calculation unit 21) that reads the teacher data from the storage unit and performs machine learning. The calculation unit performs machine learning to determine which of the j types of classification the image corresponds to, based on the teacher data, and corresponds to the k types (for example, k = 2) of the j types. Machine learning to determine which of the k types of classification an image corresponds to based on a portion of the teacher data (e.g., data sets DS1 and DS2 of the reference data 223) containing data for an image. Do. j and k are natural numbers of 2 or more. k is smaller than j. Due to these, the classification accuracy of the images by the information processing device 1 can be further improved for the classification of the k types among the j types. Therefore, images on the display device can be classified with higher precision.

Additionally, the method according to the embodiment includes a method of emphasizing the contrast of an image captured on the image display surface of a display device (for example, an image of image data included in the judgment target image data 100 or the reference data 223). Includes. The method according to the embodiment includes a step (for example, step S12) in which the information processing device (for example, the information processing device 1) performs contrast enhancement of the image. A graph (e.g., graph L31, etc.) showing the relationship between the grayscale value of a pixel included in the image before contrast enhancement and the value after contrast enhancement is a curve having two inflection points (e.g., Inflection_1, Inflection_2). am. These make it possible to set more appropriate parameters to make features caused by display defects clearer when enhancing the contrast of an image. In other words, by setting the two inflection points, features caused by display defects can be made clearer.

Additionally, a curve including one of the two inflection points (e.g., Inflection_1) and a curve including the other of the two inflection points (e.g., Inflection_2) are sigmoid curves. Accordingly, the gradation value of the contrast-enhanced image increases step by step across each inflection point, so that a contrast-enhanced image having three levels of gradation values as a whole can be obtained.

Additionally, one of the two inflection points (e.g., Inflection_1) is included in the image (e.g., judgment target image data 100 or reference data 223) in the graph (e.g., graph L31, etc.). It is at a position of a gradation value lower than the mode gradation value of a plurality of pixels included in the image of the image data, and the other of the two inflection points (for example, Inflection_2) is a plurality of pixels included in the image in the graph. It is located at a grayscale value higher than the mode grayscale value of the pixel. Therefore, by contrast enhancement, pixels with a gray-scale value lower than the mode gray-scale value of pixels included more in pixels captured in a display device of good quality and pixels with a gray-scale value higher than the mode gray-scale value can be made clearer.

In addition, the information processing device 1 of the embodiment provides an image captured on the image display surface of the display device (for example, an image of the image data included in the judgment target image data 100 or the reference data 223). It is an information processing device capable of performing contrast emphasis. The information processing device 1 includes an image processing unit (for example, an image processing unit 51) that performs contrast enhancement. A graph showing the relationship between the grayscale values of pixels included in the image before contrast enhancement and the value after contrast enhancement is a curve with two inflection points (e.g., Inflection_1, Inflection_2). With these, it is possible to provide an information processing device 1 capable of setting more appropriate parameters to make features caused by display defects clearer in enhancing the contrast of an image. In other words, by setting the two inflection points, features caused by display defects can be made clearer.

Additionally, in the above-described embodiment, j = 8 and k = 2, but this is only an example and is not limited to this. j and k are natural numbers of 2 or more, and k may be smaller than j. Inevitably, j can be said to be 3 or more. Additionally, the second inspection step and the second judgment step targeting types less than j among j types may be performed multiple times. In this case, in the second inspection step and the second judgment step, which are performed multiple times, classification limited to k types that differ from j type is performed, respectively. In addition, the second inspection performed multiple times is such that one of the second inspection steps (or second judgment steps) performed multiple times targets k types among j types, and the other targets h types among j types. The number of classifications may be different in each step and the second judgment step.

Additionally, in the above-described embodiment, the teacher data used in the second machine learning step is a part of the teacher data used in the first machine learning step, but the teacher data used in the first machine learning step and the second machine learning step The teacher data used by the staff may be separate.

In addition, in the above-described embodiment, the judgment target image data 100 and the reference data 223 are a plurality of images captured using a device that can perform imaging of a plurality of display panels at once, called a large-scale lighting inspection device. Although image data is included, this is only an example of one form of image data captured from the image display surface of the display device and is not limited to this. The image captured of the image display surface of the display device may be an image captured of the image display surface of the display device other than the display panel alone, such as an image captured of the image display surface of a smartphone.

In addition, although CNN is adopted in the above-described embodiment, this is only a specific example of a machine learning algorithm and is not limited to this. Instead of CNN in the above-described embodiment, machine learning based on another machine learning algorithm may be employed.

In addition, other functional effects brought about by the aspects described in the present embodiment are understood to be obvious from the description in the present specification, or can be appropriately imagined by those skilled in the art, as being brought about by the present disclosure.

1: Information processing device
21: calculation unit
22: memory unit
30: output unit
40: Acquisition department
50: processing unit
51: image processing unit
60: Judgment panel
62: Limited classification processing unit
70: Machine learning department
100: Image data to be judged
221: Machine learning programs
222: Image processing program
223: Reference data
224: Setting data
NN: Neural Network

Claims (6)

A display device inspection method that uses machine learning to determine which of j types of classification an image captured on the image display surface of a display device corresponds to,
A first machine, the information processing device performs machine learning to determine which of the j types of classifications an image corresponds to, based on teacher data including data of the images corresponding to the j types of classifications prepared in advance. learning steps,
The information processing device performs machine learning to determine which of the k types of classification an image corresponds to, based on teacher data including data of the image corresponding to the k type of classification among the j types. 2 Includes machine learning steps,
j and k are natural numbers greater than or equal to 2, and k is smaller than j,
Inspection method of display device. According to paragraph 1,
One type of the k types of classification is a classification of a display device that is treated as a good product that does not produce display output that is considered to be a display defect. According to claim 1 or 2,
The teacher data used in the second machine learning step is a part of the teacher data used in the first machine learning step. According to any one of claims 1 to 3,
a first inspection step in which the information processing device determines which of the j types of classification the image corresponds to after the first machine learning step and before the second machine learning step;
A first extraction step in which the information processing device extracts data of the image determined to correspond to any of the k types of classification in the first inspection step from among data of the image that has been determined in the first inspection step. class,
A display comprising a second inspection step in which the information processing device determines which of the k types of classification the image based on the data extracted in the first extraction step corresponds to after the second machine learning step. How to inspect the device. According to any one of claims 1 to 4,
a first judgment step in which the information processing device determines which of the j types of classification the data of the image different from the teacher data corresponds to;
A second extraction step in which the information processing device extracts data of the image determined to correspond to any of the k types of classification in the first judgment step from among data of the image that has been determined in the first judgment step. class,
An inspection method for a display device, comprising a second determination step in which the information processing device determines which of the k types of classification the data of the image extracted in the second extraction step corresponds to. An information processing device that uses machine learning to determine which of j types of classification an image captured on the image display surface of a display device corresponds to,
The information processing device,
a storage unit for storing teacher data including data of the image corresponding to the j type classification prepared in advance;
an arithmetic unit that reads the teacher data from the storage unit and performs machine learning;
The calculation unit is,
Based on the teacher data, machine learning is performed to determine which of the j types of classification the image corresponds to,
Based on a portion of the teacher data including data of the image corresponding to k types of classification among the j types, machine learning is performed to determine which of the k types of classification the image corresponds to,
j and k are natural numbers greater than or equal to 2, and k is smaller than j,
Information processing device.

Images