CN110570393A - A method for detecting defects in the window area of mobile phone glass cover based on machine vision - Google Patents
A method for detecting defects in the window area of mobile phone glass cover based on machine vision Download PDFInfo
- Publication number
- CN110570393A CN110570393A CN201910702333.1A CN201910702333A CN110570393A CN 110570393 A CN110570393 A CN 110570393A CN 201910702333 A CN201910702333 A CN 201910702333A CN 110570393 A CN110570393 A CN 110570393A
- Authority
- CN
- China
- Prior art keywords
- area
- defects
- mobile phone
- defect
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/94—Investigating contamination, e.g. dust
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
- G01N2021/8854—Grading and classifying of flaws
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
- G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20032—Median filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Pathology (AREA)
- Evolutionary Computation (AREA)
- Biochemistry (AREA)
- Immunology (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Analytical Chemistry (AREA)
- Chemical & Material Sciences (AREA)
- Evolutionary Biology (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Quality & Reliability (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
Abstract
本发明公开了一种基于机器视觉的手机玻璃盖板视窗区缺陷检测方法,包括如下步骤:步骤一、采集手机屏图像;步骤二、对手机屏幕区进行粗检;步骤三、通过阈值分割算法提取手机屏图像的缺陷;步骤四、利用聚类算法将密集点簇区域的散点进行连接;步骤五、利用神经网络分类器进行缺陷的分类;步骤六、提取缺陷区域的面积、长度和半径,依据检测标准进行比对;步骤七、利用深度学习分类器对缺陷进行再分类;步骤八、统计各类缺陷信息和数量。本发明检测算法遵循先粗检、后精检的原则,既能快速准确提取出各种不同型号的手机玻璃盖板屏幕区麻点、划痕、脏污、毛丝等缺陷,同时也能针对不同产品视窗区检测标准调整检测精度。
The invention discloses a method for detecting defects in the window area of a glass cover plate of a mobile phone based on machine vision, comprising the following steps: step 1, collecting an image of the mobile phone screen; step 2, performing a rough inspection on the screen area of the mobile phone; step 3, passing a threshold segmentation algorithm Extract the defects of the mobile phone screen image; Step 4, use the clustering algorithm to connect the scattered points in the dense point cluster area; Step 5, use the neural network classifier to classify the defects; Step 6, Extract the area, length and radius of the defect area , and compare according to the detection standard; Step 7, use the deep learning classifier to reclassify the defects; Step 8, count the information and quantity of various types of defects. The detection algorithm of the invention follows the principle of rough inspection first and fine inspection later, which can not only quickly and accurately extract defects such as pitting, scratches, dirt, wool and other defects in the screen area of the glass cover of various types of mobile phones, but also Adjust the detection accuracy according to the detection standards of different product window areas.
Description
技术领域technical field
本发明涉及视觉检测领域,尤其涉及一种基于机器视觉的手机玻璃盖板视窗区缺陷检测方法。The invention relates to the field of visual inspection, in particular to a method for detecting defects in the window area of a glass cover plate of a mobile phone based on machine vision.
背景技术Background technique
近年来,在5G、无线充电等技术推动下,非金属手机盖板成为主流。其中玻璃盖板有较好的力学性能和光学性能,且成本相比于陶瓷材料较低,因此备受3C产品企业青睐。然而,实际生产制造、运输等过程中,不可避免会产生麻点、划痕、脏污、崩边等缺陷。及时在生产工艺过程进行缺陷检测,可以避免工艺浪费,监控产品生产品质,从而保证高品质产品生产同时,节约生产成本。In recent years, driven by technologies such as 5G and wireless charging, non-metal mobile phone covers have become mainstream. Among them, the glass cover has better mechanical properties and optical properties, and the cost is lower than that of ceramic materials, so it is favored by 3C product companies. However, in the process of actual manufacturing, transportation, etc., defects such as pitting, scratches, dirt, and edge collapse are inevitable. Timely defect detection in the production process can avoid process waste and monitor the production quality of products, thereby ensuring the production of high-quality products and saving production costs.
目前,国内许多盖板玻璃生产厂商,仍然大量的采用人工目检的方式。机器视觉检测技术相比于人工目检过程检测效率更高、成本更低,检测标准更稳定。当前,国内许多检测设备厂商相继进行玻璃检测算法研发,但仍然存在检测过程不稳定、漏检或过检率较高等问题。At present, many domestic cover glass manufacturers still use a large number of manual visual inspection methods. Compared with the manual visual inspection process, machine vision inspection technology has higher inspection efficiency, lower cost, and more stable inspection standards. At present, many domestic testing equipment manufacturers have successively developed glass detection algorithms, but there are still problems such as unstable detection process, high missed detection or over-detection rate.
深度学习技术已经在语音识别、目标跟踪、安全监控等领域取得了广泛的应用。在机器视觉方面,深度学习在缺陷分类、定位等方面也有较大的潜力。Deep learning technology has been widely used in speech recognition, target tracking, security monitoring and other fields. In terms of machine vision, deep learning also has great potential in defect classification and localization.
发明内容SUMMARY OF THE INVENTION
为解决以上问题,本发明提供一种基于机器视觉的手机玻璃盖板视窗区缺陷检测方法,本发明适用于手机玻璃盖板中间视窗区缺陷检测,将传统检测算法与深度学习技术相结合,既能快速准确提取出各种不同型号2D或2.5D手机玻璃盖板屏幕区麻点、划痕、脏污等缺陷,同时也能针对不同产品屏幕视窗区检测标准调整检测精度。In order to solve the above problems, the present invention provides a method for detecting defects in the window area of mobile phone glass cover plates based on machine vision. It can quickly and accurately extract defects such as pitting, scratches, dirt and other defects in the screen area of 2D or 2.5D mobile phone glass cover of various models.
本发明至少通过如下技术方案之一实现。The present invention is realized by at least one of the following technical solutions.
一种基于机器视觉的手机玻璃盖板视窗区缺陷检测方法,包括如下步骤:A method for detecting defects in the window area of a glass cover of a mobile phone based on machine vision, comprising the following steps:
步骤一、采集手机屏图像;Step 1. Collect mobile phone screen images;
步骤二、对手机屏幕区进行粗检;Step 2. Perform a rough inspection on the screen area of the mobile phone;
步骤三、通过阈值分割算法提取手机屏图像的缺陷;Step 3: Extract the defects of the mobile phone screen image through a threshold segmentation algorithm;
步骤四、利用聚类算法将手机屏视窗区域上的密集点簇区域的散点进行连接;Step 4, using the clustering algorithm to connect the scattered points of the dense point cluster area on the mobile phone screen window area;
步骤五、提取缺陷的连通域特征,将连通域特征输入到神经网络分类器,利用神经网络分类器进行点状、线状和面状缺陷的分类;Step 5: Extract the connected domain features of the defects, input the connected domain features into the neural network classifier, and use the neural network classifier to classify point-like, linear and planar defects;
步骤六、对分出的点状、线状缺陷和面状缺陷区域,提取缺陷区域的面积、长度和半径,依据检测标准进行比对;Step 6: Extract the area, length and radius of the defect area for the point-like, linear-like defect and plane-like defect areas, and compare them according to the detection standard;
步骤七、利用深度学习分类器对已经分类出来的缺陷进行再分类,分类得到划痕、漂浮毛丝、浅色脏污、深色脏污四类缺陷;Step 7. Use the deep learning classifier to reclassify the classified defects, and classify four types of defects: scratches, floating hairs, light-colored dirt, and dark-colored dirt;
步骤八、统计各类缺陷信息和数量。Step 8: Statistics of various types of defect information and quantity.
进一步的,步骤一是用16K的线阵相机进行手机屏进行图像采集,图像分辨率为40000×16384,检测精度为0.005mm。Further, the first step is to use a 16K line scan camera to perform image acquisition on the mobile phone screen, the image resolution is 40000×16384, and the detection accuracy is 0.005mm.
进一步的,在进行粗检之前,选取一张无缺陷的良品手机屏图片,框选出手机屏幕区作为标准模板的感兴趣区域(ROI),离线保存标准模板文件;所述标准模板包括形状匹配模板和标准区域模板,其中,形状匹配模板主要用于与待检测图像中的手机屏进行匹配,获取得到待检测图像中手机屏的位置信息;标准区域模板主要用于与待检测图像中提取出的手机屏幕区域对齐检出缺陷;Further, before performing the rough inspection, select a picture of a defect-free good-quality mobile phone screen, frame the mobile phone screen area as the region of interest (ROI) of the standard template, and save the standard template file offline; the standard template includes shape matching. Template and standard area template, among which, the shape matching template is mainly used to match the mobile phone screen in the image to be detected, and obtain the position information of the mobile phone screen in the image to be detected; the standard area template is mainly used to extract the information from the image to be detected The mobile phone screen area is aligned to detect defects;
步骤二的对手机屏幕区进行粗检包括以下步骤:The rough inspection of the mobile phone screen area in step 2 includes the following steps:
(1)、在线检测时,读取形状匹配模板,采用基于形状的模板匹配算法,匹配获取待检测图像中手机屏的位置信息和角度信息;(1) During online detection, the shape matching template is read, and the shape-based template matching algorithm is used to match and obtain the position information and angle information of the mobile phone screen in the image to be detected;
(2)、利用阈值分割算法分割出待检测图像中手机屏视窗区域,通过获取得到的位置和角度信息,创建仿射变换矩阵;读取视窗区域的标准区域模板,利用区域仿射变换将读取的标准区域模板和待检测的视窗区域对齐;(2) Use the threshold segmentation algorithm to segment the window area of the mobile phone screen in the image to be detected, and create an affine transformation matrix by obtaining the obtained position and angle information; read the standard area template of the window area, and use the area affine transformation to read The standard area template taken is aligned with the window area to be detected;
对齐后的标准区域模板和待检测的视窗区域经过布尔运算检出是否存在缺陷;布尔运算即对区域的交、并、补运算。The aligned standard region template and the to-be-detected window region are subjected to Boolean operation to detect whether there is a defect; the Boolean operation is the operation of intersection, union and complement of the regions.
进一步的,步骤三提取手机屏图像的缺陷包括以下步骤:Further, step 3 extracting the defects of the mobile phone screen image includes the following steps:
1)、使用中值滤波进行图像预处理,以降低噪声;1), use median filtering for image preprocessing to reduce noise;
2)、将待检测的视窗区域即屏幕区切分为m×n个区域;2), dividing the window area to be detected, that is, the screen area, into m×n areas;
3)、利用高斯滤波增强算法对切分后的每个区域进行图像增强,高斯滤波增强算法为:3), use the Gaussian filter enhancement algorithm to enhance the image of each segmented area. The Gaussian filter enhancement algorithm is:
img_enhanced=round((orig-mean_gauss)*factor)+orig (1)img_enhanced=round((orig-mean_gauss)*factor)+orig(1)
公式(1)中,mean_gauss代表对图片进行高斯滤波,orig代表每幅图对应的灰度值,factor为增强因子,img_enhanced为增强后输出图像的灰度值; round()函数为取整函数;In formula (1), mean_gauss represents the Gaussian filtering of the image, orig represents the corresponding gray value of each image, factor is the enhancement factor, and img_enhanced is the gray value of the enhanced output image; the round() function is a rounding function;
4)、通过阈值分割算法对增强后的图片提取出手机屏视窗区域的缺陷。4), extract the defects in the window area of the mobile phone screen from the enhanced picture through the threshold segmentation algorithm.
进一步的,步骤四所述的聚类算法是只对满足聚类条件的密集点区域进行连接,不满足聚类条件的区域不进行聚类连接;Further, the clustering algorithm described in step 4 is to connect only the dense point regions that meet the clustering conditions, and the regions that do not meet the clustering conditions are not clustered and connected;
所述聚类条件为:The clustering conditions are:
a.聚类半径为R的圆形密集点区域;a. A circular dense point area with a clustering radius of R;
b.在圆形区域中,密集点区域面积大于密集点最小区域的面积S的区域总个数大于聚类个数N;b. In the circular area, the total number of areas where the area of the dense point area is greater than the area S of the minimum area of dense points is greater than the number of clusters N;
在半径为R的圆形区域内,若满足密集点区域面积大于设定的密集点区域最小面积S的区域个数大于N,则认为该密集点区域为密集点簇区域;利用形态学膨胀、腐蚀运算将该区域的散点连接成一个区域。In a circular area with a radius of R, if the number of areas that satisfy the requirement that the area of the dense point area is greater than the set minimum area S of the dense point area is greater than N, the dense point area is considered to be a dense point cluster area; using morphological expansion, The erosion operation connects the scatter points of this region into a region.
进一步的,所述神经网络分类器用于对点状缺陷、线状缺陷和面状缺陷的分类;Further, the neural network classifier is used to classify point defects, linear defects and planar defects;
神经网络分类器包括输入层、中间层及隐藏层;其中,输入层中神经元的个数与输入特征向量的维度相同,输出层中神经元的个数与分类输出类别相同。隐藏层把输入数据的特征,抽象到另一个维度空间,来展现其更抽象化的特征,这些特征能更好的进行线性划分。隐藏层能够提高网络的精度和表达能力。The neural network classifier includes an input layer, an intermediate layer and a hidden layer; the number of neurons in the input layer is the same as the dimension of the input feature vector, and the number of neurons in the output layer is the same as the classification output category. The hidden layer abstracts the features of the input data into another dimension space to show its more abstract features, which can be better linearly divided. Hidden layers can improve the accuracy and expressiveness of the network.
神经网络分类器进行缺陷分类主要分为两部分内容:离线训练和在线检测;The neural network classifier for defect classification is mainly divided into two parts: offline training and online detection;
在将缺陷进行分类之前,将步骤五提取的缺陷的连通域特征作为缺陷数据集;数据集保存的连通域特征分为三类,第一类为面积、中心、宽和高的基础特征;第二类为外接圆半径、圆度、紧密的形状特征;第三类为二阶矩和三阶矩的几何矩特征;Before classifying the defects, the connected domain features of the defects extracted in step 5 are used as the defect data set; the connected domain features saved in the data set are divided into three categories, the first category is the basic features of area, center, width and height; The second category is the circumscribed circle radius, roundness and compact shape features; the third category is the geometric moment features of the second and third order moments;
离线训练:设置训练参数,所述训练参数包括优化算法、学习率、学习策略、初始化策略,对缺陷数据集进行离线训练,前向传播计算损失函数,反向传播更新权重。直至损失函数达到收敛状态;离线保存训练后的神经网络分类器;Offline training: Set training parameters, including optimization algorithm, learning rate, learning strategy, initialization strategy, offline training on defective datasets, forward propagation to calculate loss function, and back propagation to update weights. Until the loss function reaches the convergence state; save the trained neural network classifier offline;
在线检测:提取缺陷的连通域特征,读取训练好的神经网络分类器,将提取的缺陷连通域特征输入分类器,得到缺陷分类结果。Online detection: Extract the connected domain features of defects, read the trained neural network classifier, and input the extracted defect connected domain features into the classifier to obtain the defect classification results.
进一步的,步骤六中所述的缺陷区域的面积是通过统计连通域中的像素个数获取,用于面状类的缺陷标准比对;缺陷区域的长度通过缺陷区域骨架化获取,用于线状类的缺陷长度标准比对;缺陷区域的半径为缺陷区域的最小外接圆半径,用于缺陷半径标准的比对。Further, the area of the defect area described in step 6 is obtained by counting the number of pixels in the connected domain, which is used for the comparison of the defect standard of the planar class; the length of the defect area is obtained by skeletonizing the defect area, which is used for the line The defect length standard comparison of the shape type; the radius of the defect area is the minimum circumcircle radius of the defect area, which is used for the comparison of the defect radius standard.
进一步的,步骤六的检测标准为:Further, the detection standard of step 6 is:
1)点状不良1) Point defects
D<0.1mm,忽略;D<0.1mm, ignore;
0.1mm≤D≤0.2mm,d≥10mm,半径为d区域范围内缺陷个数在2个以内,则表示手机屏为良品,缺陷个数大于2个则表示手机屏为不良品;0.1mm≤D≤0.2mm, d≥10mm, if the number of defects within the radius of d is less than 2, it means that the mobile phone screen is a good product, and if the number of defects is greater than 2, it means that the mobile phone screen is a defective product;
D>0.2mm,NG,表示手机屏为不良品;D>0.2mm, NG, indicating that the mobile phone screen is a defective product;
其中,D为点状缺陷最小外接圆直径,d为点与点之间距离;Among them, D is the diameter of the minimum circumscribed circle of point defects, and d is the distance between points;
2)线状不良2) Defective linearity
W<0.02mm,该缺陷忽略不计;W<0.02mm, the defect is ignored;
0.02mm≤W≤0.05mm,d≥5mm,L≤3,2个以内可以忽略该缺陷,则表示手机屏为良品;3个以内表示手机屏为不良品;0.02mm≤W≤0.05mm, d≥5mm, L≤3, if the defect can be ignored within 2, it means that the mobile phone screen is a good product; less than 3 means that the mobile phone screen is a defective product;
W>0.05mm或L>3mm,NG,表示手机屏为不良品;W>0.05mm or L>3mm, NG, indicating that the mobile phone screen is a defective product;
其中,W为线状缺陷宽度,L为线状缺陷长度,d为不同线状缺陷间距;Among them, W is the width of the linear defect, L is the length of the linear defect, and d is the distance between different linear defects;
3)面状不良3) Poor face shape
D>0.5mm或S>0.2mm2 D>0.5mm or S>0.2mm 2
其中,D为面状缺陷最大内接圆直径,S为面状缺陷面积。Among them, D is the diameter of the largest inscribed circle of the planar defect, and S is the area of the planar defect.
进一步的,所述深度学习分类器为卷积神经网络模型,该卷积神经网络模型主要包含输入层、卷积层、池化层、全连接层和输出层,其中,输入层、卷积层、池化层、全连接层和输出层依次连接;卷积层完成特征提取操作,将最后一层池化层的缺陷特征转化为一组向量作为全连接层的输入,全连接层的缺陷特征输入到输出层完成分类的任务;Further, the deep learning classifier is a convolutional neural network model, and the convolutional neural network model mainly includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, wherein the input layer, the convolutional layer , the pooling layer, the fully connected layer and the output layer are connected in turn; the convolutional layer completes the feature extraction operation, and converts the defect features of the last pooling layer into a set of vectors as the input of the fully connected layer, and the defect features of the fully connected layer Input to the output layer to complete the task of classification;
深度学习分类器进行缺陷分类主要分为两部分内容:离线训练和在线检测;Defect classification by deep learning classifier is mainly divided into two parts: offline training and online detection;
在进行缺陷分类前先制作不同类型缺陷的数据集,利用镜像、旋转对数据集中的缺陷图片进行增强;Before classifying defects, create data sets of different types of defects, and use mirroring and rotation to enhance defect pictures in the data set;
采用牛津大学视觉几何组(Visual Geometry Group)的VGG16卷积神经网络模型,设置模型结构参数、训练参数;The VGG16 convolutional neural network model of the Visual Geometry Group of Oxford University was used to set the model structure parameters and training parameters;
离线训练时,利用预训练模型对数据集进行迁移训练;训练结束后,将深度学习分类器离线保存;During offline training, use the pre-training model to perform migration training on the dataset; after training, save the deep learning classifier offline;
在线检测时,截取待分类缺陷区域对应的图片,将图片作为深度学习分类器的输入,对缺陷的进行分类。During online detection, the image corresponding to the defect area to be classified is intercepted, and the image is used as the input of the deep learning classifier to classify the defects.
进一步的,对算法参数和检测标准制作参数配置文件。参数可根据检测需求灵活调整,适用于不同型号手机玻璃盖板产品视窗区检测。Further, a parameter configuration file is made for the algorithm parameters and detection standards. The parameters can be flexibly adjusted according to the detection requirements, and it is suitable for the detection of the window area of different types of mobile phone glass cover products.
算法参数主要为各种检测算法中的关键参数,如阈值分割算法中的阈值、分块个数和聚类算法中的聚类半径等。The algorithm parameters are mainly key parameters in various detection algorithms, such as the threshold in the threshold segmentation algorithm, the number of blocks and the cluster radius in the clustering algorithm.
检测标准参数主要为点状、线状、面状检测标准中涉及的缺陷半径、长度、宽度、个数等根据检测要求设定的标准参数。The detection standard parameters are mainly the standard parameters set according to the detection requirements, such as the radius, length, width, and number of defects involved in the point, line, and plane detection standards.
进一步的,本发明遵循“先粗检、后精检”的原则。通过粗检过程,首先能对视窗区严重缺陷进行检测,通过精检过程,实现点状线状面状缺陷的分类。针对难以区分的划痕、漂浮毛丝、浅色脏污和深色脏污,采用深度学习分类器实现缺陷再分类。Further, the present invention follows the principle of "rough inspection first, then fine inspection". Through the rough inspection process, the serious defects in the window area can be detected first, and through the fine inspection process, the classification of point-like linear and planar defects can be realized. For indistinguishable scratches, floating filaments, light-colored dirt, and dark-colored dirt, a deep learning classifier is used to reclassify defects.
本发明的有益效果为:通过离线创建模板与在线检测的方式相结合,检测算法的通用性高,可适用于不同型号手机玻璃盖板的检测;遵循“先粗检,后精检”的原则,算法检测效率高;传统检测算法与深度学习技术相结合,对划痕、毛丝、深色脏污、浅色脏污等难以区分缺陷具有较好的分类效果;对算法参数和检测标准参数制作了参数配置文件,检测精度灵活可调。The beneficial effects of the invention are as follows: by combining offline creation of templates with online detection, the detection algorithm has high versatility and can be applied to the detection of glass cover plates of different types of mobile phones; the principle of “rough inspection first, then fine inspection” is followed. , the algorithm has high detection efficiency; the combination of traditional detection algorithm and deep learning technology has a good classification effect on scratches, wool, dark stains, light stains and other difficult to distinguish defects; algorithm parameters and detection standard parameters A parameter configuration file is made, and the detection accuracy is flexible and adjustable.
附图说明Description of drawings
图1是本发明一种基于机器视觉的手机玻璃盖板视窗区缺陷检测方法的检测流程示意流程图;1 is a schematic flowchart of the detection process of a method for detecting defects in the window area of a glass cover of a mobile phone based on machine vision of the present invention;
图2是本发明视窗区分块示意图;Fig. 2 is the schematic diagram of window division block of the present invention;
图3是本发明聚类条件示意图;3 is a schematic diagram of clustering conditions of the present invention;
图4是本发明聚类效果示意图;4 is a schematic diagram of the clustering effect of the present invention;
图5是本发明神经网络分类器分类示意图;Fig. 5 is the neural network classifier classification schematic diagram of the present invention;
图6是本发明神经网络分类器结构图;Fig. 6 is the neural network classifier structure diagram of the present invention;
图7a是本发明划痕的缺陷示意图;Fig. 7a is the defect schematic diagram of the scratch of the present invention;
图7b是本发明漂浮毛丝的缺陷示意图;Fig. 7b is the defect schematic diagram of the floating filament of the present invention;
图7c是本发明浅色脏污的缺陷示意图;Fig. 7c is the defect schematic diagram of the light-colored dirt of the present invention;
图7d是本发明深色脏污的缺陷示意图;Fig. 7d is the defect schematic diagram of dark stain of the present invention;
图8是本发明卷积神经网络模型结构示意图;8 is a schematic structural diagram of a convolutional neural network model of the present invention;
图9是本发明检测结果示意图;Fig. 9 is the schematic diagram of detection result of the present invention;
图10是本发明检测结果示意图。Figure 10 is a schematic diagram of the detection results of the present invention.
具体实施方式Detailed ways
下面结合具体实施例对本发明作进一步具体详细描述。The present invention will be further described in detail below in conjunction with specific embodiments.
下面结合实施例及附图对本发明作进一步详细的描述,但本发明的实施方式不限于此。The present invention will be described in further detail below with reference to the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.
如图1所示,一种基于机器视觉的手机玻璃盖板视窗区缺陷检测方法,主要粗检和精检两部分内容。粗检过程包括标准模板制作、模板匹配、仿射变换和区域比对四个关键步骤,若有严重缺陷,则进行下一步的精检。精检部分主要实现缺陷的提取和分类。待检测图像的屏幕区感兴趣区域(ROI)可以根据粗检过程匹配的位置和角度信息获取。缺陷提取过程包括图像预处理、图像分块、图像增强和阈值分割四部分。缺陷分类前,对提取得到的缺陷区域进行聚类处理;采用神经网络分类器将缺陷分为点状、线状和面状缺陷,并与对应的点状、线状、面状检测标准进行比对。采用深度学习分类器对难以区分的划伤、漂浮毛丝、浅色脏污、深色脏污进行缺陷再分类。最后,统计缺陷面积、位置、长度、宽度最小外接圆、最大内接圆等信息。As shown in Figure 1, a method for detecting defects in the window area of the glass cover of a mobile phone based on machine vision mainly includes two parts: rough inspection and fine inspection. The rough inspection process includes four key steps: standard template making, template matching, affine transformation and region comparison. If there are serious defects, the next step is fine inspection. The fine inspection part mainly realizes the extraction and classification of defects. The region of interest (ROI) of the screen area of the image to be detected can be obtained according to the position and angle information matched in the rough inspection process. The defect extraction process includes four parts: image preprocessing, image segmentation, image enhancement and threshold segmentation. Before defect classification, the extracted defect areas are clustered; the defects are divided into point, line and plane defects by neural network classifier, and compared with the corresponding point, line and plane detection standards. right. Deep learning classifiers are used to reclassify scratches, floating hairs, light-colored dirt, and dark-colored dirt that are indistinguishable. Finally, statistical information such as defect area, position, length, width minimum circumscribed circle, maximum inscribed circle and so on.
一种基于机器视觉的手机玻璃盖板视窗区缺陷检测方法,包括以下步骤:A method for detecting defects in the window area of a glass cover of a mobile phone based on machine vision, comprising the following steps:
步骤一、采用16K的线阵相机采集手机屏图像,图像分辨率为40000× 16384,检测精度可达0.005mm。Step 1. Use a 16K line scan camera to collect the image of the mobile phone screen, the image resolution is 40000 × 16384, and the detection accuracy can reach 0.005mm.
步骤二、对手机屏幕区进行粗检;Step 2. Perform a rough inspection on the screen area of the mobile phone;
在进行粗检之前,选取一张无缺陷的良品手机屏图片,框选出手机屏幕区作为标准模板的感兴趣区域(ROI)。所述标准模板主要是分为两部分:形状匹配模板和标准区域模板。其中,形状匹配模板主要用于形状匹配模板主要用于与待检测图像中的手机屏进行匹配,获取得到待检测图像中手机屏的位置信息;标准区域模板主要用于与待检测图像中提取出的手机屏幕区域对齐,检出严重缺陷。离线保存标准模板文件。Before the rough inspection, select a picture of a non-defective good mobile phone screen, and frame the mobile phone screen area as the region of interest (ROI) of the standard template. The standard template is mainly divided into two parts: a shape matching template and a standard area template. Among them, the shape matching template is mainly used to match the mobile phone screen in the image to be detected, and the position information of the mobile phone screen in the image to be detected is obtained; the standard area template is mainly used to extract the information from the image to be detected. The mobile phone screen area is aligned, and serious defects are detected. Save standard template files offline.
在线检测时,读取形状匹配模板,采用基于形状的模板匹配算法,匹配获取待检测图像中手机屏的位置信息和角度信息。During online detection, the shape matching template is read, and the shape-based template matching algorithm is used to obtain the position information and angle information of the mobile phone screen in the image to be detected.
利用阈值分割算法分割出待检测图像中手机屏视窗区域,通过获取得到的位置和角度信息,创建仿射变换矩阵;读取视窗区域的标准区域模板,利用区域仿射变换将读取的标准区域模板和待检测的视窗区域对齐;。Use the threshold segmentation algorithm to segment the window area of the mobile phone screen in the image to be detected, and create an affine transformation matrix by obtaining the obtained position and angle information; read the standard area template of the window area, and use the area affine transformation to convert the read standard area The template is aligned with the window area to be detected; .
对齐后的标准区域模板和待检测的视窗区域经过布尔运算检出是否存在严重缺陷。The aligned standard area template and the window area to be detected are checked for serious defects through Boolean operations.
步骤三、通过阈值分割算法提取手机屏图像的缺陷,具体包括以下步骤:Step 3: Extract the defects of the mobile phone screen image through a threshold segmentation algorithm, which specifically includes the following steps:
1)、使用中值滤波对图像预处理降低噪声。1), use median filter to preprocess the image to reduce noise.
如图2所示,将图片的检测区域即屏幕区切分为m×n个区域(8×6共48 个区域);由于图片尺寸较大,将屏幕区进行分块检测。分块检测一方面可以避免算法全局参数带来的缺陷漏检问题,另一方面,开启多线程,对各个小块进行并行检测,可以有效的提高检测效率。As shown in FIG. 2 , the detection area of the picture, that is, the screen area, is divided into m×n areas (8×6 total 48 areas); due to the large size of the picture, the screen area is detected in blocks. On the one hand, block detection can avoid the problem of missed detection of defects caused by the global parameters of the algorithm. On the other hand, enabling multi-threading and parallel detection of each small block can effectively improve the detection efficiency.
由于图片尺寸非常大,在缺陷提取过程中若直接用一些全局参数,如全局阈值,很容易受到光照不均等问题影响,带来缺陷漏检的问题。不同缺陷的成像对比度不同,有些缺陷,例如浅划伤成像对比度非常低,无法直接通过阈值得到,需要利用图像增强等算法进行增强。本实施例采用高斯滤波增强算法通过将图片进行高斯滤波,将滤波后的图像与原图作差达到待检测缺陷增强的效果,增强效果可以通过滤波模板大小、增强因子进行控制。可知,高斯增强可以进行局部增强,算法相较于傅里叶变换和小波变换消耗时间短,且增强效果较好。Since the image size is very large, if some global parameters, such as global threshold, are directly used in the defect extraction process, it is easy to be affected by the problem of uneven illumination, which brings about the problem of missing defect detection. The imaging contrast of different defects is different, and some defects, such as shallow scratches, have very low imaging contrast and cannot be obtained directly through the threshold value, and need to be enhanced by algorithms such as image enhancement. In this embodiment, the Gaussian filter enhancement algorithm is used to perform Gaussian filtering on the image, and the filtered image is differentiated from the original image to achieve the effect of enhancing the defects to be detected. The enhancement effect can be controlled by the size of the filter template and the enhancement factor. It can be seen that Gaussian enhancement can perform local enhancement. Compared with Fourier transform and wavelet transform, the algorithm consumes less time and has better enhancement effect.
利用高斯滤波增强算法对每个区域进行图像增强,高斯滤波增强算法为:The image enhancement is performed on each area by using the Gaussian filter enhancement algorithm. The Gaussian filter enhancement algorithm is as follows:
img_enhanced=round((orig-mean_gauss)*factor)+orig (1)img_enhanced=round((orig-mean_gauss)*factor)+orig(1)
公式(1)中,mean_gauss代表对图片进行高斯滤波,orig代表每幅图对应的灰度值,factor为增强因子,img_enhanced为增强后输出图像的灰度值; round()函数为取整函数。In formula (1), mean_gauss represents the Gaussian filtering of the image, orig represents the corresponding gray value of each image, factor is the enhancement factor, and img_enhanced is the gray value of the enhanced output image; the round() function is a rounding function.
增强后的图片,缺陷对比度得到较大的提升。The enhanced image, the defect contrast is greatly improved.
4)、通过阈值分割算法提取出缺陷。4) Defects are extracted by threshold segmentation algorithm.
步骤四、如图3和图4所示,利用聚类算法将密集点簇进行连接,具体的,将聚类半径R个像素范围内,面积大于S个像素,且小缺陷区域个数大于N个的所有区域进行聚类连接,本实施例中R=50,S=15,N=50。即在半径为50个像素的圆形区域内,若满足面积大于设定的最小面积15个像素的区域个数大于 50,则认为该部分区域为密集点簇区域。利用形态学闭运算将该区域的散点连接成一个大区域。Step 4: As shown in Figure 3 and Figure 4, the clustering algorithm is used to connect the dense point clusters. Specifically, the cluster radius is R pixels, the area is greater than S pixels, and the number of small defect areas is greater than N All regions of each are clustered and connected, in this embodiment, R=50, S=15, and N=50. That is, in a circular area with a radius of 50 pixels, if the number of areas with an area larger than the set minimum area of 15 pixels is greater than 50, it is considered that this part of the area is a dense point cluster area. Use the morphological closing operation to connect the scattered points of this area into a large area.
步骤五、提取缺陷的连通域特征,将连通域特征输入到神经网络分类器,利用神经网络分类器进行点状、线状和面状缺陷的分类,如图5所示。Step 5: Extract the connected domain features of the defects, input the connected domain features into the neural network classifier, and use the neural network classifier to classify point-shaped, linear and planar defects, as shown in Figure 5.
所述连通域特征分为:基础特征、形状特征和几何矩特征;具体如表1所示。The connected domain features are divided into basic features, shape features and geometric moment features; details are shown in Table 1.
表1连通域特征Table 1 Connected Domain Features
所述神经网络分类器进行缺陷分类主要分为两部分内容:离线训练和在线检测。Defect classification by the neural network classifier is mainly divided into two parts: offline training and online detection.
在将缺陷进行分类之前,将步骤五提取的缺陷的连通域特征作为缺陷数据集;数据集保存的连通域特征分为三类,第一类为面积、中心、宽和高的基础特征;第二类为外接圆半径、圆度、紧密的形状特征;第三类为二阶矩和三阶矩的几何矩特征。本实施例中一共选取了共22维特征。Before classifying the defects, the connected domain features of the defects extracted in step 5 are used as the defect data set; the connected domain features saved in the data set are divided into three categories, the first category is the basic features of area, center, width and height; The second category is the circumscribed circle radius, roundness, and compact shape features; the third category is the geometric moment features of the second and third order moments. In this embodiment, a total of 22-dimensional features are selected.
神经网络分类器包括输入层、中间层及隐藏层;其中,输入层中神经元的个数与输入特征向量的维度相同,输出层中神经元的个数与分类输出类别相同。隐藏层把输入数据的特征,抽象到另一个维度空间,来展现其更抽象化的特征,这些特征能更好的进行线性划分。隐藏层能够提高网络的精度和表达能力。如图6所示,为神经网络分类器的结构示意图,由22个输入层,10个隐藏层,2 个输出层构成,激活函数选用ReLU激活函数;The neural network classifier includes an input layer, an intermediate layer and a hidden layer; the number of neurons in the input layer is the same as the dimension of the input feature vector, and the number of neurons in the output layer is the same as the classification output category. The hidden layer abstracts the features of the input data into another dimension space to show its more abstract features, which can be better linearly divided. Hidden layers can improve the accuracy and expressiveness of the network. As shown in Figure 6, it is a schematic diagram of the structure of the neural network classifier, which consists of 22 input layers, 10 hidden layers, and 2 output layers, and the activation function uses the ReLU activation function;
离线训练:设置训练参数,所述训练参数包括优化算法、学习率、学习策略、初始化策略。优化算法选取动量梯度下降法,根据经验选取学习率为0.005,动量系数为0.9。权重和偏置的初始化为很小的随机数,如均值为0,方差为0.01 的高斯分布。迭代次数为2000次,每500次改变一次学习率,迭代目标为0.1。对缺陷数据集进行离线训练,前向传播计算损失函数,反向传播更新权重。直至损失函数达到收敛状态;离线保存训练后的神经网络分类器;Offline training: set training parameters, the training parameters include optimization algorithm, learning rate, learning strategy, and initialization strategy. The optimization algorithm selects the momentum gradient descent method, the learning rate is 0.005 and the momentum coefficient is 0.9 according to experience. Weights and biases are initialized to small random numbers, such as a Gaussian distribution with mean 0 and variance 0.01. The number of iterations is 2000, the learning rate is changed every 500 times, and the iteration target is 0.1. Offline training is performed on the defect dataset, forward propagation calculates the loss function, and back propagation updates the weights. Until the loss function reaches the convergence state; save the trained neural network classifier offline;
在线检测:提取缺陷的连通域特征,读取训练好的神经网络分类器,将提取的缺陷连通域特征输入分类器,得到缺陷分类结果。Online detection: Extract the connected domain features of defects, read the trained neural network classifier, and input the extracted defect connected domain features into the classifier to obtain the defect classification results.
步骤六、对分出的点状、线状和面状缺陷区域,分别提取缺陷区域的面积、长度和半径,依据检测标准进行比对;Step 6: Extract the area, length and radius of the defect area from the point-like, line-like and planar defect areas, and compare them according to the detection standard;
缺陷区域的面积通过统计连通域中的像素个数获取,用于面状类的缺陷标准比对;缺陷区域的长度通过将缺陷区域骨架化获取,用于线状类的缺陷长度标准比对;缺陷区域的半径为缺陷区域的最小外接圆半径,用于缺陷半径标准的比对。The area of the defect area is obtained by counting the number of pixels in the connected domain, which is used for the standard comparison of defects in the plane type; the length of the defect area is obtained by skeletonizing the defect area, which is used for the standard comparison of the length of the defect in the linear type; The radius of the defect area is the minimum circumcircle radius of the defect area, which is used for the comparison of defect radius standards.
依据检测标准进行比对,检测标准如下:According to the test standards, the test standards are as follows:
1)点状不良1) Point defects
D<0.1mm,忽略;D<0.1mm, ignore;
0.1mm≤D≤0.2mm,d≥10mm,半径为d区域范围内缺陷个数在2个以内则表示手机屏为良品,缺陷个数大于2个则表示手机屏为不良品;0.1mm≤D≤0.2mm, d≥10mm, if the number of defects within the radius of d is less than 2, it means that the mobile phone screen is a good product, and if the number of defects is greater than 2, it means that the mobile phone screen is a defective product;
D>0.2mm,NG表示手机屏为不良品;D>0.2mm, NG means that the mobile phone screen is a defective product;
其中,D为点状缺陷最小外接圆直径,d为点与点之间距离;Among them, D is the diameter of the minimum circumscribed circle of point defects, and d is the distance between points;
2)线状不良2) Defective linearity
W<0.02mm,忽略,不是缺陷;W<0.02mm, ignored, not a defect;
0.02mm≤W≤0.05mm,d≥5mm,L≤3,2个以内可以忽略,不是缺陷;3个以内 NG表示手机屏为不良品;0.02mm≤W≤0.05mm, d≥5mm, L≤3, within 2 can be ignored, it is not a defect; within 3, NG indicates that the mobile phone screen is a defective product;
W>0.05mm或L>3mm,NG表示手机屏为不良品;W>0.05mm or L>3mm, NG means the mobile phone screen is defective;
其中,W为线状缺陷宽度,L为线状缺陷长度,d为不同线状缺陷间距;Among them, W is the width of the linear defect, L is the length of the linear defect, and d is the distance between different linear defects;
3)面状不良3) Poor face shape
D>0.5mm或S>0.2mm2 D>0.5mm or S>0.2mm 2
其中,D为面状缺陷最大内接圆直径,S为面状缺陷面积。Among them, D is the diameter of the largest inscribed circle of the planar defect, and S is the area of the planar defect.
步骤七、利用深度学习分类器对已经分类出来的缺陷进行再分类。分类得到图7a的划痕、图7b的浮毛丝、图7c的浅色脏污和图7d的深色脏污四类缺陷;Step 7: Use the deep learning classifier to reclassify the classified defects. Four types of defects were obtained by classification: scratches in Fig. 7a, floating filaments in Fig. 7b, light stains in Fig. 7c, and dark stains in Fig. 7d;
所述深度学习分类器为卷积神经网络模型,如图8所示,该卷积神经网络模型主要包含输入层、卷积层、池化层、全连接层和输出层,其中,输入层、卷积层、池化层、卷积层、池化层、全连接层和输出层依次连接;卷积层完成特征提取操作,将最后一层池化层转化为一组向量作为全连接层的输入,全连接层到输出层完成分类的任务。The deep learning classifier is a convolutional neural network model. As shown in Figure 8, the convolutional neural network model mainly includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, wherein the input layer, Convolutional layer, pooling layer, convolutional layer, pooling layer, fully connected layer and output layer are connected in turn; The input, fully connected layer to the output layer completes the task of classification.
深度学习分类器进行缺陷分类主要分为两部分内容:离线训练和在线检测。Defect classification by deep learning classifier is mainly divided into two parts: offline training and online detection.
制作划痕、漂浮毛丝、浅色脏污、深色脏污四类缺陷的数据集。利用镜像、旋转等方式对数据集进行增强。本实施例中每类原始素材为500张,经增强10 倍后每类缺陷素材为5000张。Create a dataset of four types of defects: scratches, floating hairs, light stains, and dark stains. Enhance the dataset by mirroring, rotating, etc. In this embodiment, each type of original material is 500 pieces, and each type of defective material is 5000 pieces after being enhanced by 10 times.
采用基于卷积神经网络的深度学习分类器,设置模型结构参数、训练参数。离线训练时,利用预训练模型对数据集进行迁移训练。训练结束后,将深度学习分类器离线保存。本实施例中采用牛津大学视觉几何组(Visual Geometry Group)的VGG16卷积神经网络模型,由13个卷积层、5个池化层、3个全连接层构成。采用GPU进行并使用卷积神经网络预训练模型进行迁移训练,训练迭代次数为20000次,每32张图片为一个批次,图片初始学习率为0.005,每 5000次改变一次学习率。A deep learning classifier based on convolutional neural network is used to set the model structure parameters and training parameters. When training offline, use the pre-trained model to perform transfer training on the dataset. After training, save the deep learning classifier offline. In this embodiment, the VGG16 convolutional neural network model of the Visual Geometry Group of Oxford University is used, which is composed of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. The GPU is used for migration training and the convolutional neural network pre-training model is used for migration training. The number of training iterations is 20,000, and every 32 pictures is a batch. The initial learning rate of the picture is 0.005, and the learning rate is changed every 5,000 times.
在线检测时,截取待分类缺陷区域对应的图片,将图片作为深度学习分类器的输入,得到缺陷的分类信息。采用CPU进行测试,测试图片改为单张图片进行前向传播计算。During online detection, the picture corresponding to the defect area to be classified is intercepted, and the picture is used as the input of the deep learning classifier to obtain the classification information of the defect. The CPU is used for testing, and the test image is changed to a single image for forward propagation calculation.
步骤八、统计各类缺陷信息。Step 8: Statistics of various types of defect information.
本发明通过粗检过程,首先能对视窗区严重缺陷进行检测,通过精检过程,实现点状、线状、面状缺陷的分类。针对难以区分的划痕、漂浮毛丝、浅色脏污和深色脏污,也能实现准确的辨别。如图9和图10所示为检测结果示意图。Through the rough inspection process, the present invention can firstly detect the serious defects in the window area, and through the fine inspection process, the classification of point-like, line-like and plane-like defects can be realized. Indistinguishable scratches, floating filaments, light stains and dark stains are also accurately identified. Figures 9 and 10 are schematic diagrams of the detection results.
本发明的实施方式并不受上述实施例的限制,其他任何未背离本发明精神实质与原理下所作的改变、修饰、替代、组合、简化,均应为等效的置换方式,都包含在本发明的保护范围之内。The embodiments of the present invention are not limited by the above-mentioned examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principle of the present invention should be equivalent substitutions, and are included in this within the scope of protection of the invention.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910702333.1A CN110570393B (en) | 2019-07-31 | 2019-07-31 | A method for detecting defects in the window area of mobile phone glass cover based on machine vision |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910702333.1A CN110570393B (en) | 2019-07-31 | 2019-07-31 | A method for detecting defects in the window area of mobile phone glass cover based on machine vision |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110570393A true CN110570393A (en) | 2019-12-13 |
CN110570393B CN110570393B (en) | 2023-06-23 |
Family
ID=68773378
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910702333.1A Active CN110570393B (en) | 2019-07-31 | 2019-07-31 | A method for detecting defects in the window area of mobile phone glass cover based on machine vision |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110570393B (en) |
Cited By (40)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111060520A (en) * | 2019-12-30 | 2020-04-24 | 歌尔股份有限公司 | Product defect detection method, device and system |
CN111145258A (en) * | 2019-12-31 | 2020-05-12 | 南京埃斯顿机器人工程有限公司 | Automatic feeding and discharging method for various automobile glasses of industrial robot |
CN111179241A (en) * | 2019-12-25 | 2020-05-19 | 成都数之联科技有限公司 | Panel defect detection and classification method and system |
CN111239161A (en) * | 2020-02-28 | 2020-06-05 | 郑州旭飞光电科技有限公司 | Method and device for detecting defects of substrate glass |
CN111444921A (en) * | 2020-03-25 | 2020-07-24 | 浙江华睿科技有限公司 | Scratch defect detection method, device, computing device and storage medium |
CN111474181A (en) * | 2020-03-31 | 2020-07-31 | 成都数之联科技有限公司 | Panel defect detecting and classifying system in box forming process stage |
CN111693549A (en) * | 2020-05-14 | 2020-09-22 | 西安电子科技大学 | Method for detecting and classifying defects of mobile phone cover plate glass |
CN111696106A (en) * | 2020-07-31 | 2020-09-22 | 上海帆声图像科技有限公司 | Screen quality detection method, device and system of display equipment |
CN111709919A (en) * | 2020-06-01 | 2020-09-25 | 深圳市深视创新科技有限公司 | Glass defect visual detection algorithm |
CN111798443A (en) * | 2020-07-16 | 2020-10-20 | 佛山市南海区广工大数控装备协同创新研究院 | Method for positioning and visualizing defects by utilizing PCB defect detection system |
CN111912759A (en) * | 2020-07-15 | 2020-11-10 | 天津日博工业技术有限公司 | Method for testing air permeability of waterproof breathable film |
CN111982910A (en) * | 2020-07-06 | 2020-11-24 | 华南理工大学 | Weak supervision machine vision detection method and system based on artificial defect simulation |
CN112014413A (en) * | 2020-08-04 | 2020-12-01 | 贵州乐道科技有限公司 | Mobile phone glass cover plate window area defect detection method based on machine vision |
CN112268909A (en) * | 2020-10-20 | 2021-01-26 | 中国科学院合肥物质科学研究院 | Parameter regulation method and system for detection of surface defects and uniformity of battery pole pieces |
CN112634258A (en) * | 2020-12-30 | 2021-04-09 | 凌云光技术股份有限公司 | Automatic classification and classification method and device for defect detection based on AOI |
CN112630222A (en) * | 2020-11-24 | 2021-04-09 | 河海大学常州校区 | Mobile phone cover plate glass defect detection method based on machine vision |
CN112651972A (en) * | 2020-11-11 | 2021-04-13 | 北京平恒智能科技有限公司 | Positioning method using integral constraint of double positioning |
CN112834518A (en) * | 2021-01-06 | 2021-05-25 | 优刻得科技股份有限公司 | Particle defect detection method, system, device and medium |
CN112926439A (en) * | 2021-02-22 | 2021-06-08 | 深圳中科飞测科技股份有限公司 | Detection method and device, detection equipment and storage medium |
CN112926438A (en) * | 2021-02-22 | 2021-06-08 | 深圳中科飞测科技股份有限公司 | Detection method and device, detection equipment and storage medium |
CN113155851A (en) * | 2021-04-30 | 2021-07-23 | 西安交通大学 | Copper-clad plate surface defect visual online detection method and device based on deep learning |
CN113237888A (en) * | 2021-05-21 | 2021-08-10 | 哈尔滨工业大学 | Large-caliber optical element damage point online and offline dark field image matching method |
CN113298765A (en) * | 2021-05-19 | 2021-08-24 | 广东工业大学 | Glass panel surface defect detection method based on image processing and deep learning |
CN113504238A (en) * | 2021-06-04 | 2021-10-15 | 广东华中科技大学工业技术研究院 | Glass surface defect collecting device and detection method |
CN113538417A (en) * | 2021-08-24 | 2021-10-22 | 安徽顺鼎阿泰克科技有限公司 | Transparent container defect detection method and device based on multi-angle and target detection |
CN113609897A (en) * | 2021-06-23 | 2021-11-05 | 阿里巴巴新加坡控股有限公司 | Defect detection method and defect detection system |
CN113643224A (en) * | 2020-04-26 | 2021-11-12 | 香港科技大学 | Defect detection method and device based on semi-supervised learning |
CN113657174A (en) * | 2021-07-21 | 2021-11-16 | 北京中科慧眼科技有限公司 | Vehicle pseudo-3D information detection method and device and automatic driving system |
CN113777033A (en) * | 2021-08-18 | 2021-12-10 | 长沙长泰机器人有限公司 | Raw sliver defect detection method and device based on machine vision |
CN113850773A (en) * | 2021-09-18 | 2021-12-28 | 联想(北京)有限公司 | Detection method, device, equipment and computer readable storage medium |
CN114140679A (en) * | 2021-10-26 | 2022-03-04 | 中科慧远视觉技术(北京)有限公司 | Defect fusion method, device, recognition system and storage medium |
CN114372980A (en) * | 2022-03-21 | 2022-04-19 | 视睿(杭州)信息科技有限公司 | Industrial defect detection method and system |
CN114441547A (en) * | 2022-04-11 | 2022-05-06 | 深圳市睿阳精视科技有限公司 | Intelligent household appliance cover plate defect detection method |
CN114486916A (en) * | 2022-01-26 | 2022-05-13 | 上海电机学院 | Defect detection method of mobile phone glass cover based on machine vision |
CN114577816A (en) * | 2022-01-18 | 2022-06-03 | 广州超音速自动化科技股份有限公司 | Hydrogen fuel bipolar plate detection method |
WO2022172468A1 (en) * | 2021-02-12 | 2022-08-18 | オムロン株式会社 | Image inspection device, image inspection method, and trained model generation device |
CN115053258A (en) * | 2020-11-30 | 2022-09-13 | 京东方科技集团股份有限公司 | Display panel detection method, device and system |
CN117495856A (en) * | 2023-12-29 | 2024-02-02 | 深圳市壹倍科技有限公司 | Wafer surface detection method, device, equipment and medium based on deep learning |
CN117808737A (en) * | 2023-11-02 | 2024-04-02 | 江阴市精成数控有限公司 | Bearing defect management system |
CN119086574A (en) * | 2024-11-05 | 2024-12-06 | 清华四川能源互联网研究院 | A method for inspecting tunnels using a mother-and-child robot |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5544256A (en) * | 1993-10-22 | 1996-08-06 | International Business Machines Corporation | Automated defect classification system |
US20070177136A1 (en) * | 2006-01-23 | 2007-08-02 | Hiroyuki Nakano | Apparatus and method for inspecting defects |
CN106127779A (en) * | 2016-06-29 | 2016-11-16 | 上海晨兴希姆通电子科技有限公司 | The defect inspection method of view-based access control model identification and system |
CN106204618A (en) * | 2016-07-20 | 2016-12-07 | 南京文采科技有限责任公司 | Product surface of package defects detection based on machine vision and sorting technique |
CN107123107A (en) * | 2017-03-24 | 2017-09-01 | 广东工业大学 | Cloth defect inspection method based on neutral net deep learning |
CN108280822A (en) * | 2017-12-20 | 2018-07-13 | 歌尔科技有限公司 | The detection method and device of screen cut |
WO2018165753A1 (en) * | 2017-03-14 | 2018-09-20 | University Of Manitoba | Structure defect detection using machine learning algorithms |
CN108961238A (en) * | 2018-07-02 | 2018-12-07 | 北京百度网讯科技有限公司 | Display screen quality determining method, device, electronic equipment and storage medium |
CN109598692A (en) * | 2017-09-28 | 2019-04-09 | 南京敏光视觉智能科技有限公司 | A kind of defect inspection method based on the detection of local contrast salient region |
US20190197679A1 (en) * | 2017-12-25 | 2019-06-27 | Utechzone Co., Ltd. | Automated optical inspection method using deep learning and apparatus, computer program for performing the method, computer-readable storage medium storing the computer program,and deep learning system thereof |
CN110018178A (en) * | 2019-04-28 | 2019-07-16 | 华南理工大学 | A kind of mobile phone bend glass typical defect on-line measuring device and method |
-
2019
- 2019-07-31 CN CN201910702333.1A patent/CN110570393B/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5544256A (en) * | 1993-10-22 | 1996-08-06 | International Business Machines Corporation | Automated defect classification system |
US20070177136A1 (en) * | 2006-01-23 | 2007-08-02 | Hiroyuki Nakano | Apparatus and method for inspecting defects |
CN106127779A (en) * | 2016-06-29 | 2016-11-16 | 上海晨兴希姆通电子科技有限公司 | The defect inspection method of view-based access control model identification and system |
CN106204618A (en) * | 2016-07-20 | 2016-12-07 | 南京文采科技有限责任公司 | Product surface of package defects detection based on machine vision and sorting technique |
WO2018165753A1 (en) * | 2017-03-14 | 2018-09-20 | University Of Manitoba | Structure defect detection using machine learning algorithms |
CN107123107A (en) * | 2017-03-24 | 2017-09-01 | 广东工业大学 | Cloth defect inspection method based on neutral net deep learning |
CN109598692A (en) * | 2017-09-28 | 2019-04-09 | 南京敏光视觉智能科技有限公司 | A kind of defect inspection method based on the detection of local contrast salient region |
CN108280822A (en) * | 2017-12-20 | 2018-07-13 | 歌尔科技有限公司 | The detection method and device of screen cut |
US20190197679A1 (en) * | 2017-12-25 | 2019-06-27 | Utechzone Co., Ltd. | Automated optical inspection method using deep learning and apparatus, computer program for performing the method, computer-readable storage medium storing the computer program,and deep learning system thereof |
CN108961238A (en) * | 2018-07-02 | 2018-12-07 | 北京百度网讯科技有限公司 | Display screen quality determining method, device, electronic equipment and storage medium |
CN110018178A (en) * | 2019-04-28 | 2019-07-16 | 华南理工大学 | A kind of mobile phone bend glass typical defect on-line measuring device and method |
Non-Patent Citations (2)
Title |
---|
ZHANG,Z ET,AL.: "Specular refelction Surface Defects Detecion by using Deep Learning", 《3RD INTERNATIONAL CONFERENCE ON INFORMATION SYSTEM AND DATA MINING(ICISDM)》 * |
熊红林等: "基于多尺度卷积神经网络的玻璃表面缺陷检测方法", 《计算机集成制造系统》 * |
Cited By (54)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111179241A (en) * | 2019-12-25 | 2020-05-19 | 成都数之联科技有限公司 | Panel defect detection and classification method and system |
CN111060520A (en) * | 2019-12-30 | 2020-04-24 | 歌尔股份有限公司 | Product defect detection method, device and system |
CN111145258B (en) * | 2019-12-31 | 2023-06-02 | 南京埃斯顿机器人工程有限公司 | Method for automatically feeding and discharging various kinds of automobile glass by industrial robot |
CN111145258A (en) * | 2019-12-31 | 2020-05-12 | 南京埃斯顿机器人工程有限公司 | Automatic feeding and discharging method for various automobile glasses of industrial robot |
CN111239161A (en) * | 2020-02-28 | 2020-06-05 | 郑州旭飞光电科技有限公司 | Method and device for detecting defects of substrate glass |
CN111444921A (en) * | 2020-03-25 | 2020-07-24 | 浙江华睿科技有限公司 | Scratch defect detection method, device, computing device and storage medium |
CN111474181A (en) * | 2020-03-31 | 2020-07-31 | 成都数之联科技有限公司 | Panel defect detecting and classifying system in box forming process stage |
CN113643224A (en) * | 2020-04-26 | 2021-11-12 | 香港科技大学 | Defect detection method and device based on semi-supervised learning |
CN111693549A (en) * | 2020-05-14 | 2020-09-22 | 西安电子科技大学 | Method for detecting and classifying defects of mobile phone cover plate glass |
CN111693549B (en) * | 2020-05-14 | 2021-08-20 | 西安电子科技大学 | A kind of mobile phone cover glass defect detection and classification method |
CN111709919A (en) * | 2020-06-01 | 2020-09-25 | 深圳市深视创新科技有限公司 | Glass defect visual detection algorithm |
CN111982910A (en) * | 2020-07-06 | 2020-11-24 | 华南理工大学 | Weak supervision machine vision detection method and system based on artificial defect simulation |
CN111912759A (en) * | 2020-07-15 | 2020-11-10 | 天津日博工业技术有限公司 | Method for testing air permeability of waterproof breathable film |
CN111798443A (en) * | 2020-07-16 | 2020-10-20 | 佛山市南海区广工大数控装备协同创新研究院 | Method for positioning and visualizing defects by utilizing PCB defect detection system |
CN111696106A (en) * | 2020-07-31 | 2020-09-22 | 上海帆声图像科技有限公司 | Screen quality detection method, device and system of display equipment |
CN112014413A (en) * | 2020-08-04 | 2020-12-01 | 贵州乐道科技有限公司 | Mobile phone glass cover plate window area defect detection method based on machine vision |
CN112268909A (en) * | 2020-10-20 | 2021-01-26 | 中国科学院合肥物质科学研究院 | Parameter regulation method and system for detection of surface defects and uniformity of battery pole pieces |
CN112268909B (en) * | 2020-10-20 | 2023-10-31 | 中国科学院合肥物质科学研究院 | Parameter control method and system for surface defects and uniformity detection of battery pole pieces |
CN112651972A (en) * | 2020-11-11 | 2021-04-13 | 北京平恒智能科技有限公司 | Positioning method using integral constraint of double positioning |
CN112630222A (en) * | 2020-11-24 | 2021-04-09 | 河海大学常州校区 | Mobile phone cover plate glass defect detection method based on machine vision |
CN115053258A (en) * | 2020-11-30 | 2022-09-13 | 京东方科技集团股份有限公司 | Display panel detection method, device and system |
CN112634258A (en) * | 2020-12-30 | 2021-04-09 | 凌云光技术股份有限公司 | Automatic classification and classification method and device for defect detection based on AOI |
CN112834518A (en) * | 2021-01-06 | 2021-05-25 | 优刻得科技股份有限公司 | Particle defect detection method, system, device and medium |
WO2022172468A1 (en) * | 2021-02-12 | 2022-08-18 | オムロン株式会社 | Image inspection device, image inspection method, and trained model generation device |
CN112926439B (en) * | 2021-02-22 | 2024-08-09 | 深圳中科飞测科技股份有限公司 | Detection method and device, detection equipment and storage medium |
CN112926439A (en) * | 2021-02-22 | 2021-06-08 | 深圳中科飞测科技股份有限公司 | Detection method and device, detection equipment and storage medium |
CN112926438A (en) * | 2021-02-22 | 2021-06-08 | 深圳中科飞测科技股份有限公司 | Detection method and device, detection equipment and storage medium |
CN112926438B (en) * | 2021-02-22 | 2024-04-05 | 深圳中科飞测科技股份有限公司 | Detection method and device, detection equipment and storage medium |
CN113155851A (en) * | 2021-04-30 | 2021-07-23 | 西安交通大学 | Copper-clad plate surface defect visual online detection method and device based on deep learning |
CN113298765B (en) * | 2021-05-19 | 2023-11-07 | 广东工业大学 | Glass panel surface defect detection method based on image processing and deep learning |
CN113298765A (en) * | 2021-05-19 | 2021-08-24 | 广东工业大学 | Glass panel surface defect detection method based on image processing and deep learning |
CN113237888A (en) * | 2021-05-21 | 2021-08-10 | 哈尔滨工业大学 | Large-caliber optical element damage point online and offline dark field image matching method |
CN113504238A (en) * | 2021-06-04 | 2021-10-15 | 广东华中科技大学工业技术研究院 | Glass surface defect collecting device and detection method |
CN113504238B (en) * | 2021-06-04 | 2023-12-22 | 广东华中科技大学工业技术研究院 | Glass surface defect acquisition device and detection method |
CN113609897A (en) * | 2021-06-23 | 2021-11-05 | 阿里巴巴新加坡控股有限公司 | Defect detection method and defect detection system |
CN113657174A (en) * | 2021-07-21 | 2021-11-16 | 北京中科慧眼科技有限公司 | Vehicle pseudo-3D information detection method and device and automatic driving system |
CN113777033A (en) * | 2021-08-18 | 2021-12-10 | 长沙长泰机器人有限公司 | Raw sliver defect detection method and device based on machine vision |
CN113777033B (en) * | 2021-08-18 | 2024-08-02 | 长沙长泰机器人有限公司 | Raw strip defect detection method and device based on machine vision |
CN113538417A (en) * | 2021-08-24 | 2021-10-22 | 安徽顺鼎阿泰克科技有限公司 | Transparent container defect detection method and device based on multi-angle and target detection |
CN113850773A (en) * | 2021-09-18 | 2021-12-28 | 联想(北京)有限公司 | Detection method, device, equipment and computer readable storage medium |
CN114140679A (en) * | 2021-10-26 | 2022-03-04 | 中科慧远视觉技术(北京)有限公司 | Defect fusion method, device, recognition system and storage medium |
CN114140679B (en) * | 2021-10-26 | 2022-07-01 | 中科慧远视觉技术(北京)有限公司 | Defect fusion method, device, recognition system and storage medium |
CN114581742A (en) * | 2021-10-26 | 2022-06-03 | 中科慧远视觉技术(北京)有限公司 | Linearity-based connected domain clustering fusion method, device, system and medium |
CN114581742B (en) * | 2021-10-26 | 2023-01-24 | 中科慧远视觉技术(北京)有限公司 | Linearity-based connected domain clustering fusion method, device, system and medium |
CN114577816A (en) * | 2022-01-18 | 2022-06-03 | 广州超音速自动化科技股份有限公司 | Hydrogen fuel bipolar plate detection method |
CN114486916A (en) * | 2022-01-26 | 2022-05-13 | 上海电机学院 | Defect detection method of mobile phone glass cover based on machine vision |
CN114372980A (en) * | 2022-03-21 | 2022-04-19 | 视睿(杭州)信息科技有限公司 | Industrial defect detection method and system |
CN114441547A (en) * | 2022-04-11 | 2022-05-06 | 深圳市睿阳精视科技有限公司 | Intelligent household appliance cover plate defect detection method |
CN117808737A (en) * | 2023-11-02 | 2024-04-02 | 江阴市精成数控有限公司 | Bearing defect management system |
CN117808737B (en) * | 2023-11-02 | 2024-05-14 | 江阴市精成数控有限公司 | Bearing defect management system |
CN117495856B (en) * | 2023-12-29 | 2024-03-22 | 深圳市壹倍科技有限公司 | Wafer surface detection method, device, equipment and medium based on deep learning |
CN117495856A (en) * | 2023-12-29 | 2024-02-02 | 深圳市壹倍科技有限公司 | Wafer surface detection method, device, equipment and medium based on deep learning |
CN119086574A (en) * | 2024-11-05 | 2024-12-06 | 清华四川能源互联网研究院 | A method for inspecting tunnels using a mother-and-child robot |
CN119086574B (en) * | 2024-11-05 | 2025-01-24 | 清华四川能源互联网研究院 | A method for inspecting tunnels using a mother-and-child robot |
Also Published As
Publication number | Publication date |
---|---|
CN110570393B (en) | 2023-06-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110570393B (en) | A method for detecting defects in the window area of mobile phone glass cover based on machine vision | |
Li et al. | A novel algorithm for defect extraction and classification of mobile phone screen based on machine vision | |
Chen et al. | A light-weighted CNN model for wafer structural defect detection | |
Mei et al. | An unsupervised-learning-based approach for automated defect inspection on textured surfaces | |
CN113592845A (en) | Defect detection method and device for battery coating and storage medium | |
CN104866862B (en) | A kind of method of belt steel surface area-type defect recognition classification | |
CN113435460B (en) | A recognition method for bright crystal granular limestone images | |
CN105069807B (en) | A kind of stamped workpieces defect inspection method based on image procossing | |
CN109801286B (en) | Surface defect detection method for LCD light guide plate | |
CN108648168A (en) | IC wafer surface defects detection methods | |
CN107622277B (en) | A Bayesian Classifier-Based Defect Classification Method for Complex Surfaces | |
CN101866427A (en) | Fabric defect detection and classification method | |
CN105894036A (en) | Image feature template matching method being applied to detection of mobile phone screen defects | |
CN103913468A (en) | Multi-vision defect detecting equipment and method for large-size LCD glass substrate in production line | |
CN110827235B (en) | Steel plate surface defect detection method | |
CN105931225B (en) | A method of based on realtime graphic detection technique analyzing crystal growth form and Size Distribution | |
CN110119747A (en) | A kind of coal rock detection method based on radioscopic image | |
CN108416774A (en) | A Fabric Type Recognition Method Based on Fine-grained Neural Network | |
CN109191430A (en) | A kind of plain color cloth defect inspection method based on Laws texture in conjunction with single classification SVM | |
CN116664565A (en) | Hidden crack detection method and system for photovoltaic solar cell | |
CN106780464A (en) | A kind of fabric defect detection method based on improvement Threshold segmentation | |
CN110766664A (en) | A detection method for defective appearance of electronic components based on deep learning | |
CN107730499A (en) | A kind of leucocyte classification method based on nu SVMs | |
CN104458766A (en) | Cloth surface blemish detection method based on structure texture method | |
CN106651856A (en) | Detection method for foamed nickel surface defects |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |