CN115689970A

CN115689970A - Defect detection method and device for display panel, electronic equipment and storage medium

Info

Publication number: CN115689970A
Application number: CN202110849282.2A
Authority: CN
Inventors: 周全国; 卓磊; 李凡; 张青; 唐浩; 蒋国; 周丽佳; 降海钧; 李东阳; 徐丽蓉
Original assignee: BOE Technology Group Co Ltd
Current assignee: BOE Technology Group Co Ltd
Priority date: 2021-07-27
Filing date: 2021-07-27
Publication date: 2023-02-03

Abstract

The application discloses a defect detection method and device for a display panel, an electronic device and a storage medium. The defect detection method comprises the following steps: the method comprises the steps of obtaining an original image of a display panel attached to a U-shaped film, preprocessing the original image to obtain a preprocessed image, carrying out multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a group of multi-scale first feature maps, constructing and training the defect detection model through a deep learning multilayer convolution network based on a feature map pyramid network, and classifying panel defects according to the first feature maps. Therefore, the bad phenomenon of the display panel after the U-lami process can be accurately monitored timely, quickly and accurately, the bad monitoring timeliness of a factory is improved, and therefore follow-up related personnel can analyze the bad defects to further improve the quality of the whole product and improve the competitiveness of enterprises.

Description

Defect detection method and device for display panel, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of big data, and in particular, to a defect detection method, a defect detection apparatus, an electronic device, and a storage medium for a display panel.

Background

Generally, after the TFE process is completed on the flexible AMOLED, the cut back film (bottom film) attached to the lower surface of the panel is removed, and a U-shaped film (U-lami) is attached instead of the back film. The U-shaped film has the function of ensuring that the U-shaped film and the panel of the flexible AMOLED are not separated when the panel is bent, and meanwhile, the U-shaped film is also used as a middle bonding layer of a lower subsequent film. At present, undesirable phenomena such as foreign matters, bubbles and stabs easily occur in the process of a U-shaped membrane process, and the U-shaped membrane needs to be detected through Automatic Optical Inspection (AOI) detection, however, the existing AOI equipment can only judge whether the U-shaped membrane is qualified, and is difficult to effectively monitor specific undesirable phenomena so as to further improve.

Disclosure of Invention

The present application is directed to solving at least one of the problems in the prior art. Therefore, the application provides a defect detection method of a display panel, a defect detection device of the display panel, an electronic device and a storage medium.

The defect detection method of the display panel comprises the following steps:

acquiring an original image of the display panel attached to the U-shaped film;

preprocessing the original image to obtain a preprocessed image;

performing multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a group of multi-scale first feature maps, wherein the defect detection model is obtained by constructing and training a deep learning multilayer convolution network based on a feature map pyramid network;

and classifying the panel defects according to the first feature map.

In some embodiments, the pre-processed image includes a display area pre-processed image and a wiring area pre-processed image, the display panel is provided with a partition mark, and pre-processing the original image to obtain the pre-processed image includes:

determining panel information according to the original image;

identifying the section marks of the display panel in the original image;

dividing the original image into a display area and a wiring area according to the panel information and the partition marks;

and respectively preprocessing the display area and the wiring area to obtain a preprocessed image of the display area and a preprocessed image of the wiring area.

In some embodiments, the pre-processing the display area and the layout area to obtain a display area pre-processed image and a layout area pre-processed image respectively includes:

bilateral filtering is carried out on the display area to obtain edge complex information of the display area;

cutting the display area to remove the edge complex information of the display area;

performing image segmentation on the display area from which the edge complex information is removed to obtain a mask covered area;

and carrying out blocking processing on the mask covering area according to a preset size to obtain the display area preprocessed image.

dividing the wiring area into a foreground and a background to determine a detection range;

cutting the wiring area to remove edge complex information of the wiring area;

performing morphological processing and bilateral filtering processing on the cut wiring area;

and performing image segmentation on the processed wiring area to obtain a pre-processed image of the wiring area.

In some embodiments, the performing multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a set of multi-scale first feature maps includes:

performing feature extraction on the preprocessed image through a residual convolution network to obtain a group of second feature maps with different resolutions;

constructing a first feature map pyramid according to the group of second feature maps;

and performing up-sampling from top to bottom according to the first feature map pyramid to generate the group of multi-scale first feature maps with different resolutions.

In some embodiments, the set of multi-scale first feature maps forms a second feature map pyramid corresponding to the first feature map pyramid, and the generating the set of multi-scale first feature maps with different resolutions by upsampling according to the first feature map pyramid from top to bottom comprises:

determining a first feature map of the uppermost layer of the second feature map pyramid according to a second feature map of the uppermost layer of the first feature map pyramid;

and fusing the result of the up-sampling of the first feature map on the upper layer of the second feature map pyramid with the second feature map on the current layer of the first feature map pyramid to obtain the first feature map on the current layer.

In some embodiments, the merging the result of upsampling the first feature map of the layer above the second feature map pyramid with the second feature map of the current layer of the first feature map pyramid to obtain the first feature map of the current layer includes:

performing convolution processing on the second feature map of the first feature map pyramid current layer to reduce the number of convolution kernels of the second feature map;

performing upsampling processing on the first feature map on the upper layer of the second feature map pyramid to obtain a current upsampled feature map, so that the resolution of the current upsampled feature map is the same as that of the second feature map on the current layer;

and performing pixel superposition on the second feature map after the convolution processing and the current up-sampling feature map to obtain the first feature map of the current layer.

In some embodiments, the defect detection method includes:

establishing an algorithm model aiming at the defect type to be detected;

training the algorithm model by using the training image;

detecting a verification image by using the trained algorithm model to obtain a verification detection result so as to optimize the algorithm model;

and repeating the training steps, and determining that the algorithm model is trained to be finished to serve as the defect detection model under the condition that the accuracy of the verification detection result reaches a preset value.

In some embodiments, the defect detection method includes:

and outputting the classification result and labeling.

The application discloses display panel's defect detecting device for the panel defect detection of U type membrane laminating includes:

the acquisition module is used for acquiring an original image of the display panel attached to the U-shaped film;

the processing module is used for preprocessing the original image to obtain a preprocessed image;

the extraction module is used for carrying out multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a group of multi-scale first feature maps, and the defect detection model is constructed and trained by a deep learning multilayer convolution network based on a feature map pyramid network; and

and the classification module is used for classifying the panel defects according to the first feature map.

The electronic device of the present application includes: a processor, a memory and a program, wherein the program is stored in the memory and executed by the processor, the program comprising instructions for performing the defect detection method of any of the above.

A non-transitory computer readable storage medium of a computer program of the present application, which when executed by one or more processors, causes the processors to perform a defect detection method as described in any of the above.

In the defect detection method, the defect detection device, the electronic device and the computer storage medium of the display panel in the embodiment of the application, the original image of the U-shaped film attached to the panel is obtained, the original image is preprocessed through a visual processing algorithm to obtain a preprocessed image, then the preprocessed image is subjected to multi-scale feature extraction through a defect detection model obtained through deep learning multilayer convolution network construction training based on a feature map pyramid network, a group of multi-scale first feature maps capable of representing specific defects of the U-shaped film are obtained, and therefore the panel attached to the U-shaped film can be classified according to the first feature maps. So, can carry out the analysis according to the panel after the classification and obtain so that follow-up further improvement, effectively improve whole product quality, promote enterprise competitiveness.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow chart of a defect detection method according to some embodiments of the present application;

FIG. 2 is a block diagram of a defect detection apparatus according to some embodiments of the present application;

FIG. 3 is a block diagram of an electronic device according to some embodiments of the present application;

FIG. 4 is a schematic diagram of a pyramid network of feature maps of certain embodiments of the present application;

FIG. 5 is a schematic flow chart of a defect detection method according to some embodiments of the present application;

FIG. 6 is a schematic representation of an original image of a display panel attached to a U-shaped film according to some embodiments of the present disclosure;

7-11 are schematic flow charts of a defect detection method according to certain embodiments of the present application;

FIG. 12 is a block diagram of a defect detection apparatus according to some embodiments of the present application.

Description of the main element symbols:

the defect detection apparatus 10, the acquisition module 11, the processing module 12, the extraction module 13, the classification module 14, the creation module 15, the training module 16, the optimization module 17, the determination module 18, the electronic device 100, the processor 20, the memory 30, the program 32.

Detailed Description

Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.

Referring to fig. 1, the present application provides a defect detection method for a display panel, which is used for detecting defects of a panel attached with a U-shaped film, and the defect detection method includes the steps of:

01, acquiring an original image of the display panel attached with the U-shaped film;

02, preprocessing an original image to obtain a preprocessed image;

03, performing multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a group of multi-scale first feature maps, wherein the defect detection model is obtained by constructing and training a deep learning multilayer convolution network based on a feature map pyramid network; and

and 04, classifying the panel defects according to the first characteristic diagram.

Referring to fig. 2, the present embodiment provides a defect detecting apparatus 10 for a display panel. The defect detection device 10 comprises an acquisition module 11, a processing module 12, an extraction module 13 and a classification module 14. Step 01 may be implemented by the obtaining module 11, step 02 may be implemented by the processing module 12, step 03 may be implemented by the extracting module 13, and step 04 may be implemented by the classifying module 14.

Or, the obtaining module 11 may be configured to obtain an original image of the display panel attached to the U-shaped film;

the processing module 12 may be configured to pre-process the original image to obtain a pre-processed image.

The extraction module 13 may be configured to perform multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a group of multi-scale first feature maps, where the defect detection model is constructed and trained by a deep learning multilayer convolution network based on a feature map pyramid network.

The classification module 14 may be configured to classify the panel defect according to the first feature map.

Referring to fig. 3, the electronic device 100 of the present application further includes one or more processors 20, a memory 30; and one or more programs 32, wherein the one or more programs 32 are stored in the memory 30 and executed by the one or more processors 20, the programs 32 being executable by the processors 20 to perform the instructions of the defect detection method described above.

Referring to fig. 4, the present application further provides a non-transitory computer-readable storage medium, which stores a computer program, and when the computer program is executed by one or more processors 20, the processor 20 executes the defect detection method described above.

In the defect detection method, the detection device 10, the electronic device 100 and the storage medium according to the embodiments of the present application, the original image of the U-shaped film is obtained, the original image is preprocessed through the visual processing algorithm to obtain the preprocessed image, and then the preprocessed image is subjected to multi-scale feature extraction through the defect detection model obtained through deep learning multilayer convolution network construction training based on the feature map pyramid network to obtain a group of multi-scale first feature maps capable of representing specific defects of the U-shaped film, so that the panels attached to the U-shaped film can be classified according to the first feature maps. So, can carry out the analysis according to the panel after the classification and obtain so that follow-up further improvement, effectively improve whole product quality, promote enterprise competitiveness.

In some embodiments, the electronic device 100 may be a server, and the server may be communicatively coupled to an Automated Optical Inspection (AOI) device for inspecting the display panel attached U-shaped film. Thereby obtaining an original image of the display panel attached with the U-shaped film from the AOI equipment. The server may include a big data platform, which is a platform integrating data access, data processing, data storage, query retrieval, analysis mining, and the like, and an application interface, so that the electronic device 100 can implement the defect detection method according to the embodiment of the present disclosure.

It should be noted that the AOI device is a device for detecting common defects encountered in welding production based on optical principles. When the display panel is automatically detected, the AOI equipment automatically scans through the camera and acquires an original image of the display panel attached to the U-shaped film.

In some embodiments, the detection apparatus 10 may be part of the electronic device 100. Alternatively, the electronic device 100 comprises the detection apparatus 10.

In some embodiments, the detection apparatus 10 may be a discrete component assembled in such a way as to have the aforementioned functions, or a chip having the aforementioned functions in the form of an integrated circuit, or a piece of computer software code that causes a computer to have the aforementioned functions when run on the computer.

Referring further to fig. 3, in some embodiments, the electronic device 100 may further include a communication module, and the electronic device 100 outputs data of the detection process and/or acquires an original image of the U-shaped film to be processed by the electronic device 10 from an external device (e.g., an AOI device), for example, the communication module is connected to a database of a factory that produces display panels, so as to acquire the original image of the U-shaped film in the database.

The processor 20 may pre-process the raw image of the U-shaped membrane acquired by a visual algorithm, such as an Open CV algorithm, to obtain a pre-processed image. Among them, OPEN Source Computer Vision Library (OPEN Source Computer Vision Library) is an OPEN Source Computer Vision Library that provides many algorithm functions that efficiently implement Computer Vision algorithms. The preprocessing of the Open CV algorithm on the original image may include positioning processing and segmentation processing, that is, after the original image is obtained, the original image may be positioned through the Open CV algorithm, and then the positioned original image is segmented to obtain a preprocessed image.

The defect detection model is a mathematical model used for detecting and judging the defective defects of the image of the display panel attached to the U-shaped film, and can be established according to preset logic and a mathematical algorithm. The preset logic is a business logic, and the business logic refers to rules and processes that one entity unit should have in order to provide services to another entity unit. The mathematical algorithm may be a deep learning network algorithm based on a Feature Pyramid Network (FPN).

It should be noted that, a Feature Pyramid Network (FPN) is a network proposed in 2017, and the FPN mainly solves the multi-scale problem in object detection, and through simple network connection change, the performance of small object detection is greatly improved without basically increasing the calculated amount of the original model.

Referring to fig. 4, the feature map pyramid network includes bottom-up lines, top-down lines, and horizontal connections (horizontal connections). The bottom-up process is a common forward propagation process of the neural network, and the feature map is calculated by a convolution kernel and generally becomes smaller and smaller. From top to bottom: the top-down process is to upsample (upsample) the more abstract and semantically strong high-level feature map, and the cross-connect process is to fuse (merge) the upsampled result with a feature map of the same size generated from bottom to top. The two layers of features that are connected laterally are the same size in space, which allows the use of the underlying positioning detail information. The low resolution feature map is upsampled by a factor of 2 (for simplicity, nearest neighbor upsampling is used). The upsampled map is then combined with the corresponding bottom-up map by element addition. This process is iterated until a final first feature map is generated.

In the present application, the defect detection model may include a plurality of submodels such as a puncture detection model, a foreign object detection model, and a bubble detection model. The puncture detection model is used for processing the preprocessing image to detect whether the corresponding U-shaped film has a puncture, the foreign matter detection model is used for processing the preprocessing image to detect whether the corresponding U-shaped film has a foreign matter, and the bubble detection model is used for processing the preprocessing image to detect whether the corresponding U-shaped film has a bubble.

Further, the first feature map can be processed through an Open CV algorithm to judge and screen according to the first feature map, and whether the panel corresponding to the first feature map has defects is determined, so that classification is performed according to the defects of the panel to obtain a classification result, wherein the classification result can include but is not limited to a good panel, a puncture bad panel, a foreign matter bad panel, a bubble bad panel, a puncture and foreign matter panel, a puncture and bubble bad panel, a foreign matter and bubble bad panel, a puncture, a foreign matter and bubble bad panel and the like.

Furthermore, in some embodiments, in order to enable the relevant person to visually see the classification result of the detected panel, the obtained classification result may be labeled, and the labeled classification result may be displayed. Therefore, related people realize the specific bad phenomenon of the monitoring panel, so that the problems can be quickly found and timely improved according to the specific bad phenomenon, and the quality and yield influence caused by the bad phenomenon is reduced.

In addition, in some other embodiments, the processor 20 may further generate a bad wave notification according to the classification result and send the bad wave notification to notify the relevant personnel, for example, a large batch of bad stabbing panels occur, so that the relevant personnel can quickly view and analyze the bad wave notification, and the notification manner may be, but is not limited to, a short message notification, a telephone notification, a mail notification, and the like. For example, after monitoring the undesirable fluctuation, the processor 20 notifies the process managers in a mail manner and in a real-time alarm manner, so that the process managers can quickly check the analysis details according to the mails, and thus, the problems can be found and improved in time.

In addition, as can be seen from the above examples, the defect detection method in the present application is described by taking data obtained during the production and manufacturing process of the display panel as an example. It should be understood that the above description is only an example of the defect detection method, and does not strictly limit the object to which the defect detection method in the embodiment of the present invention is applied.

Referring to fig. 5, in some embodiments, the preprocessed images include display region preprocessed images and layout region preprocessed images, the display panel is provided with partition marks, and step 02 includes the further steps of:

021, determining panel information according to the original image;

022, identifying a partition mark of the display panel in the original image;

023, dividing the original image into a display area and a wiring area according to the panel information and the partition marks;

024 preprocessing the display area and the wiring area to obtain a display area preprocessed image and a wiring area preprocessed image.

Referring further to FIG. 2, in some embodiments, steps 021-024 can be performed by the processing module 12. That is, the processing module 12 may be configured to determine panel information from the original image and identify a partition mark of the display panel in the original image. The processing module 12 may further be configured to divide the original image into a display area and a wiring area according to the panel information and the partition marks, and perform preprocessing on the display area and the wiring area respectively to obtain a display area preprocessed image and a wiring area preprocessed image.

Referring further to FIG. 3, in some embodiments, the processor 20 may be configured to determine panel information from the original image and identify a partition marking of the display panel in the original image. The processor 20 may be further configured to divide the original image into a display area and a wiring area according to the panel information and the partition marks, and pre-process the display area and the wiring area to obtain a display area pre-processed image and a wiring area pre-processed image, respectively.

The panel information may include product size, model number, section mark of the panel, and the like.

Referring to fig. 6, specifically, the processor 20 may identify a partition mark of the display panel in the original image, divide the original image into a display area and a wiring area according to the panel information and the partition mark, and perform preprocessing on the display area and the wiring area respectively by using an Open CV algorithm to obtain a display area preprocessed image and a wiring area preprocessed image.

Therefore, the original image is divided into the display area and the wiring area according to the partition marks, and then the display area preprocessed image and the wiring area preprocessed image are obtained through preprocessing respectively, so that the display area preprocessed image and the wiring area preprocessed image are favorably and respectively processed through the defect detection model in the follow-up process, the detection accuracy of the defect detection model is improved, and meanwhile, the specific position where the bad defect exists can be further determined.

Referring to fig. 7, in some embodiments, step 024 includes the sub-steps of:

0241, bilateral filtering is carried out on the display area to obtain edge complex information of the display area;

0242, cutting the display area to remove the edge complex information of the display area;

0243, dividing the image of the display area with the edge complex information removed to obtain a mask coverage area;

0244, dividing the mask covering area into blocks according to a preset size to obtain a display area preprocessed image.

In some embodiments, sub-step 0241 to 0244 may be implemented by the processing module 12, or the processing module 12 is further configured to perform bilateral filtering on the display area to obtain edge complexity information of the display area, crop the display area to remove the edge complexity information of the display area, perform image segmentation on the display area from which the edge complexity information is removed to obtain a mask coverage area, and perform blocking processing on the mask coverage area according to a preset size to obtain a display area pre-processed image.

In some embodiments, the processor 20 may be configured to perform bilateral filtering on the display area to obtain edge complexity information of the display area, crop the display area to remove the edge complexity information of the display area, perform image segmentation on the display area from which the edge complexity information is removed to obtain a mask coverage area, and perform blocking processing on the mask coverage area according to a preset size to obtain a display area preprocessed image.

Specifically, after the display area is obtained, bilateral filtering processing can be performed on the display area through a bilateral filtering algorithm to enhance the edge characteristics of the display area, so that the comparison between a central detection area and the edge area of the display area is obvious, and then the display area is cut according to the partition marks in the display area to remove the complex information of the edge area of the display area, thereby avoiding the interference of the complex information of the edge on the positioning of the partition marks.

Furthermore, the clipped display area is divided to obtain a mask covered area with the image resolution of 15000 × 15000, and the division can be realized by adopting an area growing and dividing method. As will be understood by those skilled in the relevant art, the area growing is a process of aggregating pixels or sub-areas into larger areas according to a predefined criterion, and the basic idea is to start with a group of growing points (the growing point may be a single pixel or a small area), merge adjacent pixels or areas with similar properties to the growing point with the growing point to form a new growing point, and repeat the process until the growing point cannot grow. The similarity judgment basis of the growing points and the similar areas can be image information such as gray values, textures, colors and the like. Bilateral filtering is a nonlinear filter, which can achieve the effects of edge preservation and noise reduction smoothing. As with other filtering principles, bilateral filtering also uses a weighted average method, in which the intensity of a certain pixel is represented by a weighted average of the intensity values of the peripheral pixels, and the weighted average is based on gaussian distribution.

Furthermore, the mask coverage area is subjected to blocking processing according to a preset size (for example, 512 × 512 pixels), and a part of the mask coverage area which is smaller than the preset size is subjected to pixel overlapping processing with a part of the mask coverage area which is in front of the mask coverage area (the blocking can also be adjusted according to needs), so that N × M display area preprocessed images with preset sizes are finally generated.

Therefore, the original interference area in the display area is removed through bilateral filtering, cutting, segmentation processing and blocking processing of the display area, so that the display area preprocessed image is obtained, and the processing of the subsequent defect detection model on the display area preprocessed image is facilitated.

Referring to fig. 8, in some embodiments, step 024 includes the sub-steps of:

0245, dividing the wiring area into a foreground and a background to determine a detection range;

0246, cutting the wiring area to remove the edge complexity information of the wiring area;

0247, performing morphology processing and bilateral filtering processing on the clipped wiring area;

0248, image segmentation is carried out on the processed wiring area to obtain a wiring area pre-processing image.

In some embodiments, sub-step 02455-0248 may be implemented by the processing module 12, or the processing module 12 is further configured to divide the wire area into a foreground and a background to determine a detection range, crop the wire area to remove edge complexity information of the wire area, perform morphological processing and bilateral filtering processing on the cropped wire area, and perform image segmentation on the processed wire area to obtain a pre-processed image of the wire area.

In some embodiments, the processor 20 may be configured to divide the wire layout area into a foreground and a background to determine a detection range, crop the wire layout area to remove edge complexity information of the wire layout area, perform morphological processing and bilateral filtering processing on the cropped wire layout area, and perform image segmentation on the processed wire layout area to obtain a pre-processed image of the wire layout area.

After the wiring area is obtained, image recognition can be performed on the image of the wiring area, so that the wiring area is divided into a foreground area and a background area, wherein the foreground area is a main detection area (such as a central area of the wiring area), the background area is a secondary detection area (such as an edge area), an image recognition algorithm is not limited, and for example, division can be performed through an Otsu (OTSU) algorithm, which is an algorithm for determining an image binarization division threshold, the image is divided into a background part and a foreground part according to the gray level characteristics of the image, and the image is regarded as an optimal algorithm for selecting the threshold in image division, so that calculation is simple and is not influenced by image brightness and contrast. Furthermore, the wiring area is cut according to the division of the foreground and the background of the wiring area, and understandably, because the background area is mainly an edge area and the edge area has an image black edge, the problem of the black edge over-detection is easy to exist in the subsequent image algorithm processing process, so that the black edge boundary can be determined according to the black edge in the image and marked, and then the black edge of the wiring area is cut according to the mark.

Further, morphological processing and bilateral filtering processing are performed on the clipped wiring area, wherein the morphological processing is used for removing small black dots in the wiring area, and the bilateral filtering processing is used for enhancing edge features of the wiring area.

It should be noted that Morphology, namely mathematical Morphology (mathematical Morphology), is one of the most widely applied techniques in image processing, and is mainly used for extracting image components meaningful for expressing and describing the shape of a region from an image, so that subsequent recognition work can grasp shape features, such as boundaries and connected regions, of the most essential (most distinguishing capability — motion distinguishing) target objects. The morphological operations include erosion, dilation, and opening and closing operations.

Furthermore, the wiring area after the morphological processing and the bilateral filtering processing is subjected to image segmentation to obtain a wiring area pre-processing image. Wherein, the image segmentation can be realized by Otsu (OTSU).

Referring to fig. 9, in some embodiments, step 03 includes the sub-steps of:

031, extracting features of the preprocessed image through a residual convolution network to obtain a group of second feature maps with different resolutions;

032, constructing a first profile pyramid from the set of second profiles;

033, upsampling from top to bottom according to the first feature map pyramid to generate a set of multi-scale first feature maps with different resolutions.

With further reference to fig. 2, in some embodiments, sub-steps 031-033 may be implemented by extraction module 13. Alternatively, the extracting module 13 may be configured to perform feature extraction on the preprocessed image through a residual convolution network to obtain a set of second feature maps with different resolutions. The extraction module 13 may also be configured to construct a first feature map pyramid according to the second feature map, and perform upsampling from top to bottom according to the first feature map pyramid to generate a set of multi-scale first feature maps with different resolutions.

In some embodiments, processor 20 may be configured to perform feature extraction on the preprocessed image through a residual convolutional network to obtain a set of second feature maps with different resolutions. The processor 20 may be further configured to construct a first feature map pyramid from the second feature maps, and generate a set of multi-scale first feature maps with different resolutions by performing upsampling from top to bottom according to the first feature map pyramid.

It should be noted that the residual error network is a convolutional neural network, and the residual error network is characterized by being easy to optimize and capable of improving the accuracy rate by adding a considerable depth. The inner residual block uses jump connection, and the problem of gradient disappearance caused by depth increase in a deep neural network is relieved.

The preprocessed image comprises a display area preprocessed image and a wiring area preprocessed image. Therefore, the residual network may include a plurality of networks, the partial residual network may perform feature extraction on the display area preprocessed image to obtain a set of second feature maps with different resolutions, and the partial residual network may perform feature extraction on the wiring area preprocessed image to obtain a set of second feature maps with different resolutions, where the set of second feature maps is related to the wiring area preprocessed image.

Further, by connecting four vertices of a set of second feature images with different resolutions, a top-down feature map pyramid similar to a real pyramid can be constructed. Specifically, each second feature map in a group of second feature maps is subjected to upsampling operation to obtain a plurality of sampling layers, each adopted layer corresponds to the corresponding second feature map, then 1 × 1 convolution is used for carrying out dimensionality reduction processing on a top sampling layer, then the two sampling layers are added (corresponding elements are added), finally 3 × 3 convolution operation is carried out, then, one 3 × 3 convolution of each sampling layer is divided into two paths, 1 × 1 convolution is respectively connected for carrying out classification and regression operation to obtain candidate ROIs, the candidate ROIs are input to the sampling layers to be respectively subjected to POOL operation, finally, two 1024 layers of fully-connected network layers are connected on the basis of the previous step, then two branches are divided, and the corresponding classification layers and the regression layers are connected to generate a group of multi-scale first feature maps with different resolutions.

Referring to fig. 10, in some embodiments, a set of multi-scale first feature maps is formed in a second feature map pyramid corresponding to the first feature map pyramid, and step 033 includes the sub-steps of:

0331, determining a first feature map of the uppermost layer of the second feature map pyramid according to the second feature map of the uppermost layer of the first feature map pyramid;

0332, fusing the result of upsampling the first feature map at the layer above the second feature map pyramid with the second feature map at the current layer of the first feature map pyramid to obtain the first feature map at the current layer.

Referring further to fig. 2, in some embodiments, sub-step 0331 may be implemented by an extraction module 13. Or, the extracting module 13 may be configured to determine the first feature map of the uppermost layer of the second feature map pyramid according to the second feature map of the uppermost layer of the first feature map pyramid, and fuse the result of upsampling the first feature map of the upper layer of the second feature map pyramid and the second feature map of the current layer of the first feature map pyramid to obtain the first feature map of the current layer.

In some embodiments, the processor 20 may be configured to determine a first feature map of a top layer of the pyramid of the second feature map according to a second feature map of the top layer of the pyramid of the first feature map, and fuse a result of upsampling the first feature map of a layer above the pyramid of the second feature map and the second feature map of the current layer of the pyramid of the first feature map to obtain the first feature map of the current layer.

Specifically, the first feature map of the uppermost layer of the second feature map pyramid may be determined according to the second feature map of the uppermost layer of the first feature map pyramid, the second feature map of the current layer of the first feature map pyramid is convolved to reduce the number of convolution kernels of the second feature map, the first feature map of the previous layer of the second feature map pyramid is upsampled to obtain the current upsampled feature map, so that the resolution of the current upsampled feature map is the same as that of the second feature map of the current layer, and then the pixel superposition is performed on the convolved second feature map and the current upsampled feature map to obtain the first feature map of the current layer.

Thus, a group of multi-scale first feature maps can be obtained, and the corresponding panels can be classified according to the first feature maps.

Referring to fig. 11, in some embodiments, the defect detection method further includes:

001, aiming at the defect type to be detected, establishing an algorithm model through a deep learning multilayer convolutional neural network based on a feature map pyramid network;

002, training the algorithm model by using a training image;

003, detecting the verification image by using the trained algorithm model to obtain a verification detection result so as to optimize the algorithm model.

004, repeating the training steps, and determining that the algorithm model is trained to be used as a defect detection model under the condition that the accuracy of the detection result reaches a preset value.

Referring further to fig. 12, in some embodiments, the defect detecting apparatus may further include a creating module 15, a training module 16, an optimizing module 17, and a determining module 18. Wherein,

step 001 may be implemented by the creation module 15, step 002 may be implemented by the training module 16, the optimization module 17 may be implemented by 003, and step 004 may be implemented by the determination module.

Or the creating module 15 is configured to build an algorithm model through a deep learning multilayer convolutional neural network based on a feature map pyramid network for the defect type to be detected.

The training module 16 is used to train the algorithmic model with training images.

The optimization module 17 is configured to detect the verification image by using the trained algorithm model to obtain a verification detection result, so as to optimize the algorithm model.

The determining module 18 is configured to repeat the training steps, and determine that the training of the algorithm model is completed as the defect detection model when the accuracy of the detection result reaches the preset value.

In some embodiments, the processor 20 may be configured to establish an algorithm model by using a deep learning multilayer convolutional neural network based on a feature map pyramid network for the defect type to be detected, train the algorithm model by using a training image, and then detect a verification image by using the trained algorithm model to obtain a verification detection result so as to optimize the algorithm model. And repeating the training steps, and determining that the algorithm model is trained to be used as a defect detection model under the condition that the accuracy of the detection result reaches a preset value.

The algorithm model can comprise a plurality of algorithm models, each algorithm model corresponds to one defect type, and the training image is an original image of the display panel with the defects attached to the U-shaped film. The training image can be divided into three types of defect subimages, namely a training subimage with puncture, a training subimage with foreign matter and a training subimage with bubbles. Namely, the corresponding algorithm models are trained by using the three types of defect sub-images respectively, and the trained algorithm models are used for detecting the verification images to obtain verification detection results so as to optimize the algorithm models. And under the condition that the accuracy of the detection result is verified to reach a preset value, determining that the algorithm model is completely trained to serve as a defect detection model.

The types of defects to be detected may include puncture defects, foreign matter defects, and bubble defects.

Further, before the algorithm model is trained through the training image, each type of defect sub-image can be divided into a display area defect sub-image and a wiring area defect sub-image. Furthermore, the defects in the display area defect sub-image and the wiring area defect sub-image are marked to determine the type, size, position and the like of the defects. In addition, considering that the number of the collected pictures is small at present, in some examples, data amplification can be performed on each type of defect sub-image, so that the number of each type of defect is increased, and the accuracy of algorithm training is ensured.

The training detection result obtained by training the algorithm model through the training image is compared with the actual detection result, so that the algorithm model is continuously adjusted and optimized, the training detection result is close to the actual detection result, when the accuracy and the over-detection rate of the training detection result reach the standard, the algorithm model can be considered to reach the standard, and the trained algorithm model can be used as a defect detection model for defect detection.

Therefore, the algorithm model can be trained by utilizing the training image and the training recognition result to obtain a trained defect detection model, and the trained defect detection model can be obtained according to the defect detection module.

In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware or any other combination. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a Digital Video Disc (DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), among others.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A defect detection method of a display panel is used for detecting defects of a panel attached with a U-shaped film, and is characterized by comprising the following steps:

acquiring an original image of the display panel attached to the U-shaped film;

preprocessing the original image to obtain a preprocessed image;

performing multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a group of multi-scale first feature maps, wherein the defect detection model is constructed and trained by a deep learning multilayer convolution network based on a feature map pyramid network;

and classifying the panel defects according to the first feature map.

2. The defect detection method of claim 1, wherein the pre-processed image comprises a display area pre-processed image and a routing area pre-processed image, the display panel is provided with partition marks, and pre-processing the original image to obtain a pre-processed image comprises:

determining panel information according to the original image;

identifying the section marks of the display panel in the original image;

3. The method of claim 2, wherein the pre-processing the display area and the wire area to obtain a display area pre-processed image and a wire area pre-processed image respectively comprises:

carrying out image segmentation on the display area from which the edge complex information is removed to obtain a mask covered area;

and carrying out blocking processing on the mask coverage area according to a preset size to obtain the display area preprocessed image.

4. The method of claim 2, wherein the pre-processing the display area and the wire area to obtain a display area pre-processed image and a wire area pre-processed image respectively comprises:

cutting the wiring area to remove edge complex information of the wiring area;

performing morphological processing and bilateral filtering processing on the wiring area after cutting;

5. The defect detection method of claim 1, wherein the obtaining a set of multi-scale first feature maps by performing multi-scale feature extraction on the preprocessed image through a defect detection model comprises:

and performing up-sampling according to the first feature map pyramid from top to bottom to generate the group of multi-scale first feature maps with different resolutions.

6. The defect detection method of claim 5 wherein the set of multi-scale first feature maps forms a second feature map pyramid corresponding to the first feature map pyramid, and wherein upsampling from the top-down according to the first feature map pyramid to generate the set of multi-scale first feature maps with different resolutions comprises:

7. The defect detection method of claim 6, wherein the fusing the result of upsampling the first feature map of the second feature map pyramid upper layer with the second feature map of the first feature map pyramid current layer to obtain the first feature map of the current layer comprises:

performing convolution processing on the second feature map of the current layer of the pyramid of the first feature map so as to reduce the number of convolution kernels of the second feature map;

performing upsampling processing on the first feature map at a layer above the second feature map pyramid to obtain a current upsampled feature map, so that the resolution of the current upsampled feature map is the same as that of the second feature map at the current layer;

8. The defect detection method of claim 1, wherein the defect detection method comprises:

establishing an algorithm model aiming at the defect type to be detected;

training the algorithm model by using a training image;

9. The defect detection method of claim 1, wherein the defect detection method comprises:

and outputting the classification result and labeling.

10. The utility model provides a defect detecting device of display panel for the panel defect detection of U type membrane laminating, its characterized in that, detecting device includes:

11. An electronic device, comprising a processor, a memory, and a program, wherein the program is stored in the memory and executed by the processor, the program comprising instructions for performing the defect detection method of any of claims 1-9.

12. A non-transitory computer-readable storage medium of a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the defect detection method of any one of claims 1-9.