CN117788790A - Material installation detection method, system, equipment and medium for general scene - Google Patents
Material installation detection method, system, equipment and medium for general scene Download PDFInfo
- Publication number
- CN117788790A CN117788790A CN202311726126.2A CN202311726126A CN117788790A CN 117788790 A CN117788790 A CN 117788790A CN 202311726126 A CN202311726126 A CN 202311726126A CN 117788790 A CN117788790 A CN 117788790A
- Authority
- CN
- China
- Prior art keywords
- calibration
- image
- scene
- target
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 171
- 239000000463 material Substances 0.000 title claims abstract description 109
- 238000009434 installation Methods 0.000 title claims abstract description 97
- 238000000034 method Methods 0.000 claims abstract description 61
- 238000000605 extraction Methods 0.000 claims abstract description 7
- 230000011218 segmentation Effects 0.000 claims description 22
- 230000008569 process Effects 0.000 claims description 21
- 230000019771 cognition Effects 0.000 claims description 20
- 239000013077 target material Substances 0.000 claims description 19
- 230000001149 cognitive effect Effects 0.000 claims description 13
- 238000005286 illumination Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 11
- 230000000007 visual effect Effects 0.000 claims description 11
- 238000007781 pre-processing Methods 0.000 claims description 8
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 6
- 238000011426 transformation method Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 230000000877 morphologic effect Effects 0.000 claims description 4
- 238000012549 training Methods 0.000 abstract description 10
- 230000002452 interceptive effect Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 14
- 238000011897 real-time detection Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000003562 lightweight material Substances 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of material installation detection, in particular to a general scene-oriented material installation detection method, a general scene-oriented material installation detection system, a general scene-oriented material installation detection device and a general scene-oriented material installation detection medium, which comprise the steps of carrying out matching calibration on a material installation scene image and a calibration template in a scene recognition library to obtain a target detection image, searching a matching template matched with the target detection image from the scene recognition library, and carrying out feature extraction on the target detection image according to the matching template to obtain the level key features of the target detection image; and calculating the level key feature similarity between the target detection image and the matching template by using a linear weighting method, so as to generate a material correct installation detection result. According to the invention, the interactive detection is carried out by adopting the method of automatically selecting the template, the operation cost is reduced, the requirement on the richness of the data is extremely low, when a new part is introduced into a detection task, the real-time target classification is carried out by only needing a small amount of data without collecting a large amount of data again and manually marking for training, so that the rapid detection is realized and the accuracy is improved.
Description
Technical Field
The invention relates to the technical field of material installation detection, in particular to a general scene-oriented material installation detection method, a system, equipment and a medium.
Background
In an industrial scene, for tasks such as material installation, defect detection and installation behavior supervision, the production efficiency of a workpiece is affected due to low efficiency of manual inspection, and different people judge defect standards and installation behaviors of sheet metal parts, so that the workpiece has variability, meanwhile, along with the continuous increase of labor cost, the production cost is also increased, the computing capacity of a computer is continuously enhanced under the rapid development of big data age and computer graphic card technology, the artificial intelligence is rapidly developed, and the application of computer vision in industrial detection is increasingly wide.
Object Detection (Object Detection) is one of the core problems in the computer vision field, aims at identifying all interested Object objects in an image and determining the category and the position of the Object objects, is always the most challenging problem in the computer vision field due to the diversity of appearance, shape and gesture of various objects, and the interference of factors such as illumination and shielding, and is mainly divided into four tasks of classification, detection, positioning and segmentation, and the currently mainstream Object Detection technology mainly comprises a neural network represented by Yolo, nanodet, mobileNet, and the network has a certain discrimination capability on different types of workpieces by learning an example sample acquired by a high-precision industrial camera under a specific scene, can achieve real-time Detection, makes state judgment of current material placement or installation, and is combined with a display system to remind workers to perform inspection and error correction.
Although the accuracy of target detection is higher, the model body is larger, the model is not suitable for being transplanted to a mobile terminal or embedded equipment, the characteristics of materials detected in a fixed scene are single, the quantity is small, the pixels are high, the network is not required to have strong fitting capability on the characteristics of the materials, particularly in certain scenes, the similarity between part of parts and the materials is strong, the same small part can appear for a plurality of times, the judgment on the installation condition of the same part in different positions becomes a very difficult problem, and the situations of misjudgment, lower accuracy and the like can be caused by using a neural network training sample.
Meanwhile, the prior art lacks a guide for object-specific learning, namely object detection, but is realized through understanding the overall concept of big data rather than learning the inherent mode and the inherent attribute of the object, so that the algorithm is difficult to adapt to the old object which appears in a new posture or form or the new object which is never seen by the algorithm, and the new element except a learned sample is jumped out, in addition, due to the diversity and the unknown of an industrial scene, the situation that the variety of a certain part is too rare in the market and the sample is insufficient exists, when the new part is introduced into a detection task, the data needs to be acquired again for manual marking and then training, however, the actual scene of a factory is complex and changeable, so that a worker is not realistic to collect the data according to the network training standard, and the problems of unnecessary time and memory consumption, low migration of the algorithm and the like are caused.
Disclosure of Invention
The invention provides a material installation detection method, a system, equipment and a medium for a general scene, which solve the technical problems that the existing material installation detection depends on data driving, has poor adaptability to new elements, has poor mobility and cannot effectively cope with complex scenes with diversity and unknowns of factories.
In order to solve the technical problems, the invention provides a material installation detection method, a system, equipment and a medium for a general scene.
In a first aspect, the present invention provides a method for detecting installation of materials in a general scene, the method comprising the steps of:
acquiring a material installation scene image, and carrying out matching calibration on the material installation scene image and a calibration template in a scene cognition library established in advance to obtain a target calibration image;
preprocessing the target calibration image to obtain a target detection image, searching a matching template matched with the target detection image from the scene recognition library, and acquiring a target material detection position of the target detection image according to the matching template;
based on the target material detection position, extracting features of the target detection image by using a mixed feature extractor to obtain hierarchical key features of the target detection image; the hierarchical key features include multi-level component features, material features, and shape features; the matching template comprises a hierarchy key feature corresponding to the matching template;
Calculating the level key feature similarity between the target detection image and the matching template by using a linear weighting method;
and comparing the similarity of the key features of the hierarchy with a set similarity threshold value to generate a material correct installation detection result.
In a further embodiment, the process of creating the matching template includes:
acquiring a scene image of a matching template to be acquired, and carrying out matching calibration on the scene image of the matching template to be acquired by using a calibration template to obtain a calibration scene image of the matching template to be acquired;
performing multi-level disassembly on the calibration scene image of the matching template to be acquired, and establishing a multi-level cognitive structure of the calibration scene image of the matching template to be acquired;
extracting features of the multi-level cognitive structure by using a mixed feature extractor to obtain basic features corresponding to different levels of cognitive structures;
and combining the basic features by utilizing linear combination to obtain the level key features of the scene images of the matched templates to be acquired, and constructing the matched templates according to the level key features.
In a further embodiment, the step of performing multi-level disassembly on the to-be-acquired matching template calibration scene image to establish a multi-level cognitive structure of the to-be-acquired matching template calibration scene image includes:
Performing frame selection on the calibration scene image of the matching template to be acquired by utilizing a semantic segmentation model to obtain a target candidate frame and coordinates thereof, and performing component multi-level disassembly on the target candidate frame to generate a target component;
wherein the basic features include visual word features extracted from the target component, color features, texture features, and edge features extracted from the target candidate frame.
In a further embodiment, the semantic segmentation model is a SAM semantic segmentation model; the step of obtaining a target candidate frame and coordinates thereof specifically comprises the following steps of:
dividing the calibration scene image of the template to be acquired by using a semantic division model to obtain a mask image after division;
performing morphological processing on the mask map to obtain an optimized mask map;
and constructing a minimum circumscribed parallelogram according to the optimized mask map, and performing frame selection on the calibration scene image of the matching template to be acquired by utilizing the minimum circumscribed parallelogram to obtain a target candidate frame and coordinates thereof.
In a further embodiment, the hybrid feature extractor includes a LAB color space feature extractor, a texture filter bank feature extractor, a directional gradient histogram space pyramid feature extractor, and a Canny edge detector feature extractor.
In a further embodiment, the step of matching and calibrating the material installation scene image with calibration templates in a pre-established scene recognition library to obtain a target calibration image specifically includes:
searching a calibration template matched with the material installation scene image from a pre-established scene recognition library;
performing calibration operation on the material installation scene image by using the calibration template, comparing the calibrated material installation scene image with a preset calibration standard reference object, judging whether the calibrated material installation scene image reaches the calibration standard, and if so, acquiring a target calibration image containing a target material detection position according to the calibrated material installation scene image; otherwise, the calibration operation is carried out again until the calibration standard is reached; the calibration operations include position calibration, illumination calibration, and resolution calibration.
In a further embodiment, the position calibration employs a corner detection method and an affine transformation method;
the illumination calibration adopts a gamma correction method;
the resolution calibration uses an affine transformation method.
In a second aspect, the present invention provides a general-purpose scenario-oriented material installation detection system, the system comprising:
The image calibration module is used for acquiring a material installation scene image, and carrying out matching calibration on the material installation scene image and a calibration template in a scene cognition library established in advance to acquire a target calibration image;
the template matching module is used for preprocessing the target calibration image to obtain a target detection image, searching a matching template matched with the target detection image from the scene recognition library, and acquiring a target material detection position of the target detection image according to the matching template;
the feature extraction module is used for extracting features of the target detection image by utilizing a mixed feature extractor based on the target material detection position to obtain level key features of the target detection image; the hierarchical key features include multi-level component features, material features, and shape features; the matching template comprises a hierarchy key feature corresponding to the matching template;
and the detection judging module is used for calculating the similarity of the level key features between the target detection image and the matching template by using a linear weighting method, and comparing the similarity of the level key features with a set similarity threshold value to generate a material correct installation detection result.
In a third aspect, the present invention also provides a computer device, including a processor and a memory, where the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the computer device performs steps for implementing the method.
In a fourth aspect, the present invention also provides a computer readable storage medium having stored therein a computer program which when executed by a processor performs the steps of the above method.
The invention provides a general scene-oriented material installation detection method, a system, equipment and a medium, wherein the method is characterized in that a material installation scene image is matched and calibrated with a calibration template in a scene cognition library established in advance to obtain a target detection image, a matching template matched with the target detection image is searched from the scene cognition library, and a mixed feature extractor is utilized to extract features of the target detection image according to the matching template to obtain the level key features of the target detection image; hierarchical key features include multi-level component features, material features, and shape features; calculating the level key feature similarity between the target detection image and the matching template by using a linear weighting method; and comparing the similarity of the key features of the hierarchy with a set similarity threshold value to generate a material correct installation detection result. Compared with the prior art, the method aims at static material installation conditions under various unknown scenes, combines a general algorithm and different cognitions, provides a lightweight algorithm to realize real-time detection of material installation states, can realize rapid detection rate by a small amount of semantic feature information of a multi-level structure, does not need to acquire a large amount of data again and manually mark for training, and ensures that the algorithm has generalization, flexibility and mobility and can be highly suitable for various complex industrial scenes.
Drawings
FIG. 1 is a schematic flow chart of a general scene-oriented material installation detection method provided by an embodiment of the invention;
FIG. 2 is a schematic diagram of a general system for detecting material installation according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a calibration template creation process according to an embodiment of the present invention;
FIG. 4 is a block diagram of an automatic calibration flow of a camera according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a matching template establishment process according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a hierarchical structure of matching templates according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a matching algorithm implementation flow provided by an embodiment of the present invention;
FIG. 8 is a schematic diagram of an overall frame for static scene material installation detection provided by an embodiment of the invention;
FIG. 9 is a block diagram of a general scenario-oriented material installation detection system provided by an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following examples are given for the purpose of illustration only and are not to be construed as limiting the invention, including the drawings for reference and description only, and are not to be construed as limiting the scope of the invention as many variations thereof are possible without departing from the spirit and scope of the invention.
Referring to fig. 1, an embodiment of the present invention provides a method for detecting material installation oriented to a general scene, as shown in fig. 1, including the following steps:
s1, acquiring a material installation scene image, and carrying out matching calibration on the material installation scene image and a calibration template in a scene cognition library established in advance to obtain a target calibration image.
Because the facing industrial scene is a more general scene and does not have the excellent environments of camera fixation, light source fixation, no personnel interference and no physical layer interference elimination during standard industrial detection, the automatic calibration function of the excellent camera is very critical and necessary for the more general industrial scene (such as a manual assembly line process), therefore, in the embodiment, in most assembly line tasks existing in the industrial scene, the camera is controlled to be in a overlooking machine position so as to ensure real-time data acquisition, and simultaneously avoid interference with the operation of workers, meanwhile, the embodiment introduces the idea of a scene cognition library, namely the understanding and knowledge of people on correct conditions during detection, the scene cognition library further guides the direction and the emphasis point of algorithm detection through the scene cognition library, and as shown in fig. 2, the scene cognition library of the whole material installation and detection general system consists of a calibration template and a matching template, the contents of the two parts are determined by specific scene specific tasks, and the overlooking machine position facing the standard industrial detection template and the matching template can be acquired through the overlooking machine position in a detection box; the method is characterized in that the method is used for selecting proper machine position height, light source configuration and component angle acquisition by means of expert knowledge in the face of general universal industrial scenes 'calibration templates' and 'matching templates'.
Fig. 3 is a diagram of a calibration template establishing process provided in this embodiment, the calibration template is one of the modes of detecting a scene recognition library through participation in a guiding algorithm, and is applied to automatic calibration of a camera, in the calibration process, the calibration template provides standards and directions to help calculate correct calibration parameters, so as to exclude interference of complex environments, in order to ensure that calibration of the camera is accurate, in this embodiment, parameters such as proper camera height, light source configuration, component angle and the like are selected through expert knowledge when the calibration template is constructed, in obtaining the parameters, 8-10 photos are continuously taken in 3 seconds in this embodiment, then the photos are named and saved to positions corresponding to corresponding scene folders in the scene recognition library, in the actual detection process, the calibration template is used as a guiding or denoising means for calibrating an input image, so that the illumination environment, camera distance and visual field position of the calibrated input image before formal detection are consistent with the calibration template.
In a real-time detection process, capturing a material installation scene image through a camera, triggering a calibration template which is matched with the material installation scene image in the same scene from a pre-established scene cognition library after the material installation scene image is captured, and performing calibration operation on the input material installation scene image according to the calibration template, wherein the calibration operation comprises position calibration, illumination calibration and resolution calibration; in the position calibration, for the offset visual field, the embodiment preferentially adopts a corner detection method, an affine transformation method and the like to lead the visual field of the camera to be positive, and if the offset is too large and no target object exists in the visual field, the correction cannot be completed through the image processing technology, and at the moment, an alarm is triggered to remind a worker to align the camera; in illumination calibration, for the conditions of different illumination scenes, the embodiment adopts a gamma correction method, and different brightness levels are unified into a certain brightness through gamma change, so that the influence caused by illumination is reduced; in the resolution calibration, the object sizes are different due to the deviation of the camera distance and the difference of the distances from the object, so the embodiment uses the affine transformation method to change the field of view into a fixed size for subsequent detection, and fig. 4 is a flow chart of the automatic calibration of the camera.
After the input material installation scene image is calibrated, locking calibration parameters and sending out a calibrated signal, comparing the calibrated material installation scene image with a preset calibration standard reference object, judging whether the calibrated material installation scene image reaches the calibration standard, and if yes, obtaining a target calibration image containing a target material detection position according to the calibrated material installation scene image; otherwise, the calibration operation is carried out again until the calibration standard is reached, and a target calibration image is obtained.
S2, preprocessing the target calibration image to obtain a target detection image, searching a matching template matched with the target detection image from the scene recognition library, and acquiring a target material detection position of the target detection image according to the matching template.
Since the size and shape of the actual object will deform during the calibration process, the present embodiment needs to perform preprocessing on the target calibration image, where the preprocessing operation is to normalize the target calibration image to make it into a target detection image with a fixed size, so as to facilitate subsequent detection, after data preprocessing, the object will match with a preset matching template, and if the object matches with a correct material, it is determined that the installation condition is correct, otherwise, it is wrong.
In this embodiment, the matching template is another way of the scene recognition library to participate in the guided algorithm detection, which is used for comparing in the detection process, to detect whether the material is correctly installed, and considering that the detected material is often changed, and the worker may lack expertise or lack expertise to select a suitable matching template, while the existing template matching methods all use global matching, the global matching is difficult to set a suitable discrimination threshold when facing smaller errors, the threshold is small and easy to be interfered, the threshold is large and lacks sufficient discrimination, so the embodiment designs a semi-automatic template collecting method, defines a hierarchical structure for maximizing semantic information of the matching template, establishes a three-level cognitive structure of scenes, objects and parts & attributes for the matching template to be input into the scene recognition library, captures the semantic information by using semantic segmentation models, feature extraction and other ways, and converts the global matching for the objects into three finer dimensions of parts, materials and shapes for matching through multi-level disassembly, thereby enhancing the discrimination of features, in this embodiment, the building process of the template specifically includes:
Acquiring a scene image of a template to be acquired, and carrying out matching calibration on the scene image of the template to be acquired by using a calibration template to obtain a calibration scene image of the template to be acquired under the same condition as the calibration template; then carrying out multi-level disassembly on the calibration scene image of the matching template to be acquired, and establishing a multi-level cognitive structure of the calibration scene image of the matching template to be acquired; then, extracting features of the multi-level cognitive structure by using a mixed feature extractor to obtain basic features corresponding to different levels of cognitive structures; and combining the basic features by utilizing linear combination to obtain the level key features of the scene images of the matched templates to be acquired, and constructing the matched templates according to the level key features.
The step of performing multi-level disassembly on the to-be-acquired matching template calibration scene image to establish a multi-level cognitive structure of the to-be-acquired matching template calibration scene image comprises the following steps: and carrying out frame selection on the calibration scene image of the matching template to be acquired by utilizing a semantic segmentation model to obtain a target candidate frame and coordinates thereof, and carrying out multi-level component disassembly on the target candidate frame to generate a target component.
In this embodiment, the basic features include visual word features extracted from the target component, color features, texture features and edge features extracted from the target candidate frame, specifically, as shown in fig. 5, a matching template construction process is mainly completed by a semi-automatic matching template acquisition module, and a user determines a component area to be acquired by clicking, so that the method automatically completes frame selection operation on the matching template calibration scene image to be acquired by using a semantic segmentation model to obtain the target candidate frame and coordinates thereof, thereby preventing non-professional acquisition of staff; then clicking for multiple times in the target candidate frame area after frame selection is completed, completing disassembly of the target candidate frame component parts to generate target component parts, and then sending the target candidate frame automatically selected and the disassembled component parts into a mixed feature extractor, wherein the mixed feature extractor extracts visual word features from the target component parts, color features, texture features and edge features from the target candidate frame, and the target component parts obtain component features through the visual word features; the target candidate frame generates corresponding material characteristics and shape characteristics through color characteristics, texture characteristics and edge characteristics, the embodiment utilizes linear combination to combine basic characteristics such as visual word characteristics, color characteristics, texture characteristics and edge characteristics to obtain the level key characteristics of the scene image of the matching template to be acquired, and finally, the embodiment constructs the matching template according to the level key characteristics and the coordinates of the target candidate frame, so that the matching template of the current scene recognition library is constructed.
FIG. 6 is a schematic diagram of a hierarchical structure of matching templates, in FIG. 6, a scene represents a scene where a matching template stored in a scene recognition library is located, and the scene includes an object and scene information; the object represents a main object to be detected in the scene and is used for participating in a subsequent matching algorithm; the components represent the composition of an object, such as the screws and the louvers of the heat sink shown in fig. 6.
The feature of any detected object can be regarded as being composed of three parts and attribute features, namely a part feature, a material feature and a shape feature, wherein the three features are formed by linear combination of four basic features through a certain rule, the basic features are obtained by extracting a mixed feature extractor, the mixed feature extractor comprises an LAB color space feature extractor (LAB values), a texture filter bank feature extractor (texton filterbank), a directional gradient histogram space pyramid feature extractor (HOG spatial pyramid) and a Canny edge detector feature extractor (Canny edge detector), the LAB values represent color histograms of calculated images, and the color features are represented by using statistical information of image pixels; texton filterbank represents a texton filter bank, using a set of filters to obtain respective ones of the different textures for representing the texture features; HOG spatial pyramid, calculating an HOG feature pyramid of the image, and using statistical characteristics of multi-azimuth gradient information to describe input components so as to obtain visual word features; canny edge detector the canny edge feature of the image is computed and the shape feature of the image is obtained.
Because the SAM semantic segmentation model is a large segmentation model and has good zero sample segmentation performance, namely, a language, a position and other prompts are provided for the unseen class, the SAM semantic segmentation model is preferentially selected in the embodiment, the matching template acquisition construction is realized based on the SAM semantic segmentation model, meanwhile, in order to reduce noise during SAM semantic segmentation and enhance the robustness of a system, and meanwhile, the system storage is reduced, the embodiment performs optimization operation on the segmentation result, in order to reduce noise during SAM semantic segmentation, after the segmentation operation, smaller segmentation areas are removed, larger segmentation areas are reserved, the adjacent areas are connected, morphological optimization is performed on a mask map, meanwhile, in order to reduce the robustness of the storage and the enhancement system, the mask after segmentation optimization is not directly used, and a minimum external parallelogram is constructed according to the mask, so that the subsequent matching anti-interference capability is enhanced.
Dividing the calibration scene image of the template to be acquired by using a semantic division model to obtain a mask image after division;
performing morphological processing on the mask map to obtain an optimized mask map;
and constructing a minimum circumscribed parallelogram according to the optimized mask map, and performing frame selection on the calibration scene image of the matching template to be acquired by utilizing the minimum circumscribed parallelogram to obtain a target candidate frame and coordinates thereof.
S3, based on the target material detection position, extracting features of the target detection image by using a mixed feature extractor to obtain level key features of the target detection image; the hierarchical key features include multi-level component features, material features, and shape features; the matching template includes hierarchical key features corresponding thereto.
In the real-time detection process, a trigger signal enters a scene recognition library after the image calibration is completed to search a matching template of a corresponding scene, the target detection image obtains a target material detection position according to the searched matching template, and then the target detection image obtains multi-level component characteristics, material characteristics and shape characteristics by using a mixed characteristic extractor which is the same as that in the scene recognition library based on the target material detection position, so that the level key characteristics of the target detection image are obtained.
S4, calculating the level key feature similarity between the target detection image and the matching template by using a linear weighting method.
S5, comparing the similarity of the key features of the hierarchy with a set similarity threshold value to generate a material correct installation detection result.
The traditional template matching algorithm is mainly based on pixel units or simple gradient characteristics to compare images, wherein gradients are differences among pixels, the algorithm shows higher sensitivity to the direction of pixel change, the gradient characteristics are particularly important when the direction problems (such as assembly errors and assembly inclinations) are required to be concerned in the process of material assembly and the like, in addition, the gradients describe the changes of the pixels, so that the influence of illumination on all pixels in the detection process is reduced, however, information loss is caused only by using the gradients or the pixels, color information cannot be well saved by using gradient information, the gradient information is focused on the whole image, and the algorithm is insensitive to partial changes, and on the other hand, the influence of the interference factors such as picture offset, illumination and shadow is easily influenced only by using the pixel information.
Therefore, in order to more effectively solve the problem of image matching, the embodiment provides a cognition-based multi-level feature rich in semantic information, and the features of a scene cognition library, which are multi-level and rich in semantic information, are utilized for comparison.
As shown in fig. 7, in this embodiment, whether the target detection image is correctly installed or not is determined by the detection determining module, specifically, the target detection image is compared with three hierarchical key features of the matching template, the hierarchical key feature similarity is obtained by calculating by a linear weighting method of feature similarity, finally, a reasonable threshold is set according to the task precision requirement, the hierarchical key feature similarity is compared with the set similarity threshold to generate a material correct installation detection result, after detection is completed, on one hand, the assembly condition of the component in the scene is encoded and saved, if three positions are to be checked, 1 indicates that the assembly is correct, and 0 indicates that the assembly is wrong, for example: 010 shown in fig. 7 indicates that the second position is correct and the rest is wrong; on the other hand, the detection result is displayed on the monitoring device, the correct result is indicated by a green box, and the incorrect result is indicated by a red box.
Fig. 8 is a schematic diagram of a static scene material installation detection overall framework, and compared with a complex training process of a neural network and high requirements on a dataset, a scene recognition library is designed by using a template matching algorithm based on a fixed position and a determined target posture in a scene, a human recognition is introduced into a material installation detection method for targeted learning, a corresponding hierarchical structure is established through understanding objects, features rich in physical significance are constructed, feature precision of current template matching is optimized, requirements of different tasks can be perfectly met, the algorithm is relatively simple and easy to realize, and meanwhile, the algorithm has high accuracy and robustness, and can effectively solve the problem of image matching in different tasks.
According to the embodiment, the material installation and detection problems are converted into simpler classification problems, a lightweight algorithm is used for real-time target classification, for the problem requirements under different scenes, such as heat sink placement position detection, chip position detection, screw installation detection and the like, rapid detection rate can be realized through a small amount of data, meanwhile, the embodiment adopts a method of automatically selecting templates for character interaction, operation cost is reduced, popularization to all workers is convenient, the requirement on the richness of data is extremely low, the diversity and the unknown of industrial scenes are low, when a certain part on the market is sparse in variety and insufficient in sample, when a new part is introduced into a detection task, a large amount of data is not required to be acquired again and manual labeling is not required for training, classification and regression functions can be realized through a small amount of training, and the state judgment of current material placement or installation is detected in real time.
The embodiment of the invention provides a general scene-oriented material installation detection method, which comprises the steps of carrying out matching calibration on a material installation scene image and a calibration template in a scene recognition library to obtain a target detection image, searching a matching template matched with the target detection image from the scene recognition library, and acquiring a target material detection position of the target detection image according to the matching template; based on the detection position of the target material, extracting the characteristics of the target detection image by using a mixed characteristic extractor to obtain the level key characteristics of the target detection image; calculating the level key feature similarity between the target detection image and the matching template by using a linear weighting method; and comparing the similarity of the key features of the hierarchy with a set similarity threshold value to generate a material correct installation detection result. Compared with the existing neural network detection method, the lightweight material installation detection method provided by the embodiment carries out real-time target classification through a small amount of data, the algorithm is relatively simple and easy to implement, the problem of image matching in different tasks can be effectively solved, meanwhile, a scene cognition library is introduced, and a general algorithm is combined with different cognition methods to realize rapid detection and improve accuracy, so that the algorithm has generalization, flexibility and mobility and is highly suitable for industrial scenes with complex conditions.
It should be noted that, the sequence number of each process does not mean that the execution sequence of each process is determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.
In one embodiment, as shown in fig. 9, an embodiment of the present invention provides a general-purpose scenario-oriented material installation detection system, the system including:
the image calibration module 101 is used for acquiring a material installation scene image, and carrying out matching calibration on the material installation scene image and a calibration template in a scene recognition library established in advance to obtain a target calibration image;
the template matching module 102 is configured to pre-process the target calibration image to obtain a target detection image, search a matching template matching the target detection image from the scene recognition library, and obtain a target material detection position of the target detection image according to the matching template;
the feature extraction module 103 is configured to perform feature extraction on the target detection image by using a hybrid feature extractor based on the target material detection position, so as to obtain a level key feature of the target detection image; the hierarchical key features include multi-level component features, material features, and shape features; the matching template comprises a hierarchy key feature corresponding to the matching template;
And the detection judging module 104 is used for calculating the similarity of the level key features between the target detection image and the matching template by using a linear weighting method, and comparing the similarity of the level key features with a set similarity threshold value to generate a detection result for correct installation of the material.
For specific limitation of a general-purpose scenario-oriented material installation detection system, reference may be made to the above limitation of a general-purpose scenario-oriented material installation detection system method, which is not repeated herein. Those of ordinary skill in the art will appreciate that the various modules and steps described in connection with the embodiments disclosed herein may be implemented as hardware, software, or a combination of both. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The embodiment of the invention provides a general scene-oriented material installation detection system, which carries out matching calibration on a material installation scene image and a calibration template in a scene cognition library established in advance through an image calibration module to obtain a target calibration image; searching a matching template matched with the target detection image from the scene recognition library through a template matching module; extracting features of the target detection image through a feature extraction module to obtain hierarchical key features of the target detection image; and calculating the level key feature similarity between the target detection image and the matching template through the detection judgment module, and determining a material correct installation detection result according to the level key feature similarity. Compared with the prior art, the method has the advantages that a scene cognition library is introduced, the cognition of human beings is introduced into a material installation detection method to learn in a targeted manner, a corresponding hierarchical structure is established, the feature rich in physical meaning is constructed, the feature precision of the current template matching is optimized, and in an industrial scene with diversity and unknown property, when a new part is introduced into a detection task, a large amount of data is not required to be collected again and manually marked for training, real-time target classification is performed by using a lightweight algorithm, and high-efficiency detection rate is realized by a small amount of data.
FIG. 10 is a diagram of a computer device including a memory, a processor, and a transceiver connected by a bus, according to an embodiment of the present invention; the memory is used to store a set of computer program instructions and data and the stored data may be transferred to the processor, which may execute the program instructions stored by the memory to perform the steps of the above-described method.
Wherein the memory may comprise volatile memory or nonvolatile memory, or may comprise both volatile and nonvolatile memory; the processor may be a central processing unit, a microprocessor, an application specific integrated circuit, a programmable logic device, or a combination thereof. By way of example and not limitation, the programmable logic device described above may be a complex programmable logic device, a field programmable gate array, general purpose array logic, or any combination thereof.
In addition, the memory may be a physically separate unit or may be integrated with the processor.
It will be appreciated by those of ordinary skill in the art that the structure shown in fig. 10 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have the same arrangement of components.
In one embodiment, an embodiment of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above-described method.
According to the general scene-oriented material installation detection method, system, equipment and medium provided by the embodiment of the invention, the lightweight algorithm is adopted for real-time target classification, high-efficiency real-time detection efficiency can be realized only by a small amount of data, the acquisition of a large amount of data is not needed, the training is performed by manual labeling, meanwhile, the multi-level structure of the scene recognition library is introduced to construct the feature with rich physical meaning, the feature can be learned in a targeted manner, the template matching precision is improved, the rapid detection and the accuracy are realized, and the method has the advantages of low cost and easiness in popularization.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.
Those skilled in the art will appreciate that implementing all or part of the above described embodiment methods may be accomplished by way of a computer program stored on a computer readable storage medium, which when executed, may comprise the steps of embodiments of the methods described above.
The foregoing examples represent only a few preferred embodiments of the present application, which are described in more detail and are not thereby to be construed as limiting the scope of the invention. It should be noted that modifications and substitutions can be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and substitutions should also be considered to be within the scope of the present application. Therefore, the protection scope of the patent application is subject to the protection scope of the claims.
Claims (10)
1. The universal scene-oriented material installation detection method is characterized by comprising the following steps of:
acquiring a material installation scene image, and carrying out matching calibration on the material installation scene image and a calibration template in a scene cognition library established in advance to obtain a target calibration image;
Preprocessing the target calibration image to obtain a target detection image, searching a matching template matched with the target detection image from the scene recognition library, and acquiring a target material detection position of the target detection image according to the matching template;
based on the target material detection position, extracting features of the target detection image by using a mixed feature extractor to obtain hierarchical key features of the target detection image; the hierarchical key features include multi-level component features, material features, and shape features; the matching template comprises a hierarchy key feature corresponding to the matching template;
calculating the level key feature similarity between the target detection image and the matching template by using a linear weighting method;
and comparing the similarity of the key features of the hierarchy with a set similarity threshold value to generate a material correct installation detection result.
2. The method for detecting material installation oriented to a general scene as claimed in claim 1, wherein the process of establishing the matching template comprises:
acquiring a scene image of a matching template to be acquired, and carrying out matching calibration on the scene image of the matching template to be acquired by using a calibration template to obtain a calibration scene image of the matching template to be acquired;
Performing multi-level disassembly on the calibration scene image of the matching template to be acquired, and establishing a multi-level cognitive structure of the calibration scene image of the matching template to be acquired;
extracting features of the multi-level cognitive structure by using a mixed feature extractor to obtain basic features corresponding to different levels of cognitive structures;
and combining the basic features by utilizing linear combination to obtain the level key features of the scene images of the matched templates to be acquired, and constructing the matched templates according to the level key features.
3. The method for detecting material installation oriented to a general scene as claimed in claim 2, wherein the step of performing multi-level disassembly on the calibration scene image of the matching template to be acquired, and establishing a multi-level cognitive structure of the calibration scene image of the matching template to be acquired comprises:
performing frame selection on the calibration scene image of the matching template to be acquired by utilizing a semantic segmentation model to obtain a target candidate frame and coordinates thereof, and performing component multi-level disassembly on the target candidate frame to generate a target component;
wherein the basic features include visual word features extracted from the target component, color features, texture features, and edge features extracted from the target candidate frame.
4. A general scene oriented material installation detection method according to claim 3, wherein the semantic segmentation model is a SAM semantic segmentation model; the step of obtaining a target candidate frame and coordinates thereof specifically comprises the following steps of:
dividing the calibration scene image of the template to be acquired by using a semantic division model to obtain a mask image after division;
performing morphological processing on the mask map to obtain an optimized mask map;
and constructing a minimum circumscribed parallelogram according to the optimized mask map, and performing frame selection on the calibration scene image of the matching template to be acquired by utilizing the minimum circumscribed parallelogram to obtain a target candidate frame and coordinates thereof.
5. The general-purpose scene-oriented material installation detection method as claimed in claim 1, wherein: the hybrid feature extractor includes a LAB color space feature extractor, a texture filter bank feature extractor, a directional gradient histogram space pyramid feature extractor, and a Canny edge detector feature extractor.
6. The method for detecting the installation of materials in a general scene according to claim 1, wherein the step of matching and calibrating the installation scene image of the materials with calibration templates in a pre-established scene recognition library to obtain a target calibration image specifically comprises:
Searching a calibration template matched with the material installation scene image from a pre-established scene recognition library;
performing calibration operation on the material installation scene image by using the calibration template, comparing the calibrated material installation scene image with a preset calibration standard reference object, judging whether the calibrated material installation scene image reaches the calibration standard, and if so, acquiring a target calibration image containing a target material detection position according to the calibrated material installation scene image; otherwise, the calibration operation is carried out again until the calibration standard is reached; the calibration operations include position calibration, illumination calibration, and resolution calibration.
7. The general-purpose scene-oriented material installation detection method as claimed in claim 6, wherein: the position calibration adopts a corner detection method and an affine transformation method;
the illumination calibration adopts a gamma correction method;
the resolution calibration uses an affine transformation method.
8. A universal scene oriented material installation detection system, the system comprising:
the image calibration module is used for acquiring a material installation scene image, and carrying out matching calibration on the material installation scene image and a calibration template in a scene cognition library established in advance to acquire a target calibration image;
The template matching module is used for preprocessing the target calibration image to obtain a target detection image, searching a matching template matched with the target detection image from the scene recognition library, and acquiring a target material detection position of the target detection image according to the matching template;
the feature extraction module is used for extracting features of the target detection image by utilizing a mixed feature extractor based on the target material detection position to obtain level key features of the target detection image; the hierarchical key features include multi-level component features, material features, and shape features; the matching template comprises a hierarchy key feature corresponding to the matching template;
and the detection judging module is used for calculating the similarity of the level key features between the target detection image and the matching template by using a linear weighting method, and comparing the similarity of the level key features with a set similarity threshold value to generate a material correct installation detection result.
9. A computer device, characterized by: comprising a processor and a memory, the processor being connected to the memory, the memory being for storing a computer program, the processor being for executing the computer program stored in the memory to cause the computer device to perform the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized by: the computer readable storage medium having stored therein a computer program which, when executed, implements the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311726126.2A CN117788790A (en) | 2023-12-14 | 2023-12-14 | Material installation detection method, system, equipment and medium for general scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311726126.2A CN117788790A (en) | 2023-12-14 | 2023-12-14 | Material installation detection method, system, equipment and medium for general scene |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117788790A true CN117788790A (en) | 2024-03-29 |
Family
ID=90393823
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311726126.2A Pending CN117788790A (en) | 2023-12-14 | 2023-12-14 | Material installation detection method, system, equipment and medium for general scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117788790A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118397255A (en) * | 2024-06-26 | 2024-07-26 | 杭州海康威视数字技术股份有限公司 | Method, device and equipment for determining analysis area and intelligently analyzing analysis area |
| CN119810628A (en) * | 2024-12-12 | 2025-04-11 | 贵州大学 | A fast indoor object recognition method based on scene understanding |
| CN119922408A (en) * | 2025-03-28 | 2025-05-02 | 天翼视联科技有限公司 | Calibration method, device and storage medium for AI algorithm-controlled camera |
-
2023
- 2023-12-14 CN CN202311726126.2A patent/CN117788790A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118397255A (en) * | 2024-06-26 | 2024-07-26 | 杭州海康威视数字技术股份有限公司 | Method, device and equipment for determining analysis area and intelligently analyzing analysis area |
| CN119810628A (en) * | 2024-12-12 | 2025-04-11 | 贵州大学 | A fast indoor object recognition method based on scene understanding |
| CN119922408A (en) * | 2025-03-28 | 2025-05-02 | 天翼视联科技有限公司 | Calibration method, device and storage medium for AI algorithm-controlled camera |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3806064B1 (en) | Method and apparatus for detecting parking space usage condition, electronic device, and storage medium | |
| CN117788790A (en) | Material installation detection method, system, equipment and medium for general scene | |
| CN109253722B (en) | Monocular distance measuring system, method, equipment and storage medium fusing semantic segmentation | |
| CN112001298B (en) | Pedestrian detection method, device, electronic equipment and storage medium | |
| CN108875600A (en) | A kind of information of vehicles detection and tracking method, apparatus and computer storage medium based on YOLO | |
| US20130243343A1 (en) | Method and device for people group detection | |
| CN105574550A (en) | Vehicle identification method and device | |
| CN114627089B (en) | Defect identification method, defect identification device, computer equipment and computer readable storage medium | |
| CN112802049B (en) | A method and system for constructing a household object detection data set | |
| CN112541403B (en) | Indoor personnel falling detection method by utilizing infrared camera | |
| CN113537026B (en) | Method, device, equipment and medium for detecting graphic elements in building plan | |
| CN111967527B (en) | A method and system for identifying peony varieties based on artificial intelligence | |
| CN116993653B (en) | Camera lens defect detection method, device, equipment, storage medium and product | |
| CN116740758A (en) | Bird image recognition method and system for preventing misjudgment | |
| CN112634368A (en) | Method and device for generating space and OR graph model of scene target and electronic equipment | |
| CN114119695A (en) | Image annotation method and device and electronic equipment | |
| CN114219753A (en) | Power equipment surface defect detection method based on deep learning and terminal | |
| CN111652168B (en) | Group detection method, device, equipment and storage medium based on artificial intelligence | |
| CN108573226A (en) | Keypoint localization method of Drosophila larval body segments based on cascaded pose regression | |
| CN114612563A (en) | Aviation cable automatic plug-in method, system and storage medium | |
| CN114742145A (en) | Performance test method, device and equipment of target detection model and storage medium | |
| Sun et al. | An infrared-optical image registration method for industrial blower monitoring based on contour-shape descriptors | |
| CN111738264A (en) | An intelligent collection method of display panel data of equipment room equipment | |
| CN117423149A (en) | Intelligent attendance checking system based on face recognition and statistical method thereof | |
| CN113537197B (en) | Meter automatic modeling method based on machine vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |