CN110047147A - A kind of 3D point cloud processing method, device, system and computer storage medium - Google Patents
A kind of 3D point cloud processing method, device, system and computer storage medium Download PDFInfo
- Publication number
- CN110047147A CN110047147A CN201910282317.1A CN201910282317A CN110047147A CN 110047147 A CN110047147 A CN 110047147A CN 201910282317 A CN201910282317 A CN 201910282317A CN 110047147 A CN110047147 A CN 110047147A
- Authority
- CN
- China
- Prior art keywords
- mould group
- point cloud
- camera mould
- roi
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000003860 storage Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims description 45
- 230000015654 memory Effects 0.000 claims description 31
- 239000000284 extract Substances 0.000 claims description 17
- 239000011159 matrix material Substances 0.000 claims description 15
- 230000007704 transition Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims 1
- 238000004519 manufacturing process Methods 0.000 abstract description 7
- 230000008569 process Effects 0.000 description 9
- 230000001360 synchronised effect Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000004026 adhesive bonding Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000005291 magnetic effect Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 239000000306 component Substances 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 238000009434 installation Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000008358 core component Substances 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 239000011248 coating agent Substances 0.000 description 1
- 238000000576 coating method Methods 0.000 description 1
- 230000005294 ferromagnetic effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a kind of 3D point cloud processing method, device, system and computer storage mediums, it include: 2D camera mould group, 3D camera mould group and the camera bearing support being fixed on the 2D camera mould group and the 3D camera mould group above the accurate carrying platform in X/Y plane, wherein, which comprises collaboration calibration is carried out to 2D camera mould group and 3D camera mould group；The 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain the 2D image and 3D point cloud of target workpiece；Based on the 2D characteristics of image that 2D camera mould group obtains, 2D ROI is extracted；By the 2D ROI backprojection to 3D point cloud space, the 3D ROI that projection meets 2D ROI is obtained；A cloud computing is carried out to the 3D ROI, 3D parameter required for obtaining.Through the embodiment of the present invention, the 3D point cloud quantity of calculating under the premise of not influencing accuracy of identification, can be substantially reduced, arithmetic speed is improved, improves 3D point cloud application real-time performance, computing resource is saved, optimizes real-time processing speed in actual production.
Description
Technical field
The present invention relates to field of machine vision, in particular to a kind of 3D point cloud processing method, device, system and computer are deposited
Storage media.
Background technique
3D point cloud processing technique is just more and more being paid attention in recent years.Integer data is used with same visual field 2D image
The mode of record is compared, and 3D point cloud is due to introducing the three-dimensional information of target workpiece and being stored using floating number, required meter
Calculation ability is substantially improved, time-consuming significant extension, brings obstruction to the real-time application of technology.
In practical applications, area-of-interest (ROI, Region Of Interest) is often entire 3D point cloud image
In a part.But ROI directly is filtered out in 3D point cloud data, it needs to calculate entire point cloud traversal once, and by
In the only three-dimensional information of X/Y/Z, it is difficult to quickly be positioned by the characteristics of image on surface.
So related 3D point cloud processing technique, during processing in the presence of processing points are more, flop operating speed is slow, calculates
The problem of force request height, real-time characteristic difference.
So with the rapid development of industry, being badly in need of one kind now under the premise of not influencing accuracy of identification, capable of significantly subtracting
The 3D point cloud quantity calculated needed for few improves arithmetic speed, improves 3D point cloud application real-time performance, saves the 3D point of computing resource
The processing method of cloud.
Summary of the invention
It is situated between in view of this, the embodiment of the present invention provides a kind of 3D point cloud processing method, device, system and computer storage
Matter, can be under the premise of not influencing accuracy of identification, and the 3D point cloud quantity calculated needed for substantially reducing improves arithmetic speed, changes
Kind 3D point cloud application real-time performance, saves computing resource, optimizes the real-time processing speed of equipment in actual production, significantly mention
Rise 3D treatment efficiency.
It is as follows that the present invention solves technical solution used by above-mentioned technical problem:
According to an aspect of the present invention, a kind of 3D point cloud processing method provided is applied to 3D point cloud processing unit, institute
Stating 3D point cloud processing unit includes: 2D camera mould group, 3D camera mould group and by the 2D camera mould group and the 3D camera mould
Group is fixed on the camera bearing support above the accurate carrying platform in X/Y plane, wherein the described method includes:
Collaboration calibration is carried out to 2D camera mould group and 3D camera mould group；
The 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain the 2D image and 3D point cloud of target workpiece；
Based on the 2D characteristics of image that 2D camera mould group obtains, 2D ROI is extracted；
By the 2D ROI backprojection to 3D point cloud space, the 3D ROI that projection meets 2D ROI is obtained；
A cloud computing is carried out to the 3D ROI, 3D parameter required for obtaining.
It is described that collaboration calibration is carried out to 2D camera mould group and 3D camera mould group in a possible design；Include:
The 2D camera mould group and 3D camera mould group are individually demarcated respectively；
Collaboration calibration is carried out to 2D camera mould group and 3D camera mould group, the picture position 2D is established and turns to 3D point cloud X/Y plane
Change matrix T23.
In a possible design, the 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain target workpiece
2D image and 3D point cloud；Include:
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece；
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece.
In a possible design, the 2D characteristics of image obtained based on 2D camera mould group extracts 2D ROI；Packet
It includes:
According to the inspected feature of target workpiece, 2D image is handled, extract 2D ROI, obtains it on the 2 d image
Location information MROI-2D.
It is described by the 2D ROI backprojection to 3D point cloud space in a possible design, it obtains projection and meets
The 3D ROI of 2D ROI；Include:
According to the transition matrix T23, by 2D ROI backprojection to 3D point cloud space, corresponding 3D ROI is in X/Y plane
On range be MROI-2D-3D；
By in 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D ROI.
It is described that a cloud computing is carried out to 3D ROI in a possible design, 3D parameter required for obtaining；Include:
Point cloud in 3D ROI is calculated, 3D parameter required for obtaining.
According to another aspect of the present invention, a kind of 3D point cloud processing unit provided is applied to a kind of 3D point cloud and handles
Method, the 3D point cloud processing unit, comprising: 2D camera mould group, 3D camera mould group, camera bearing support and accurate carrying are flat
Platform, in which:
The precision carrying platform, is located in X/Y plane, in the X/Y plane precise motion, for carrying target workpiece；
The 2D camera mould group, is fixed on the camera bearing support, and shooting direction is perpendicular to the accurate carrying
Platform, for acquiring the image information for obtaining the target workpiece；
The 3D camera mould group, is fixed on the camera bearing support, with 2D camera mould group relative position, claps
Direction is taken the photograph perpendicular to the accurate carrying platform, and the direction XY is identical as the 2D camera mould group setting, obtains institute for acquiring
State the 3D point cloud of target workpiece；
The camera bearing support, for the 2D camera mould group and the 3D camera mould group to be fixed on the precision and hold
Above carrying platform, and the relative positional relationship of fixed the 2D camera mould group and the 3D camera mould group.
According to another aspect of the present invention, a kind of 3D point cloud processing system provided, comprising: memory, processor and
It is stored in the computer program that can be run on the memory and on the processor, the computer program is by the processing
Device realizes a kind of the step of described 3D point cloud processing method provided in an embodiment of the present invention when executing.
According to another aspect of the present invention, a kind of computer readable storage medium provided, which is characterized in that the meter
The program of 3D point cloud processing method is stored on calculation machine readable storage medium storing program for executing, the program of the 3D point cloud processing method is held by processor
The step of described a kind of 3D point cloud processing method provided in an embodiment of the present invention is realized when row.
Compared with prior art, a kind of 3D point cloud processing method, device, system and computer provided in an embodiment of the present invention
Storage medium, comprising: 2D camera mould group, 3D camera mould group and fix the 2D camera mould group and the 3D camera mould group
Camera bearing support above the accurate carrying platform in X/Y plane, wherein the described method includes: to 2D camera mould group and 3D
Camera mould group carries out collaboration calibration；The 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain the 2D image of target workpiece
And 3D point cloud；Based on the 2D characteristics of image that 2D camera mould group obtains, 2D ROI is extracted；By the 2D ROI backprojection to 3D point
Cloud space obtains the 3D ROI that projection meets 2D ROI；A cloud computing is carried out to the 3D ROI, the ginseng of 3D required for obtaining
Number.Through the embodiment of the present invention, by integrating the image processing algorithm of 2D and 3D, first through 2D image zooming-out 2D ROI, then reversely
The point cloud spatial extraction 3D ROI for projecting 3D does a cloud computing, to substantially reduce under the premise of not influencing accuracy of identification
The 3D point cloud quantity of required calculating improves arithmetic speed, improves the effect of operation efficiency, saves computing resource, improves 3D point cloud
Using real-time performance, the real-time processing speed of equipment in actual production is optimized, 3D treatment efficiency is obviously improved.
Detailed description of the invention
Fig. 1 provides a kind of structural schematic diagram of 3D point cloud processing unit for the embodiment of the present invention；
Fig. 2 provides a kind of flow diagram of 3D point cloud processing method for the embodiment of the present invention；
Fig. 3 provides a kind of structural schematic diagram of 3D point cloud processing unit for the embodiment of the present invention；
Fig. 4 provides a kind of 3D point cloud processing unit treated the shoes wall 3D rendering letter of sole gluing for the embodiment of the present invention
Cease the schematic diagram extracted；
Fig. 5 provides a kind of flow diagram of 3D point cloud processing method for the embodiment of the present invention；
Fig. 6 provides a kind of structural schematic diagram of 3D point cloud processing unit for the embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
In order to be clearer and more clear technical problems, technical solutions and advantages to be solved, tie below
Drawings and examples are closed, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only
To explain the present invention, it is not intended to limit the present invention.
In subsequent description, it is only using the suffix for indicating such as " module ", " component " or " unit " of element
Be conducive to explanation of the invention, itself there is no a specific meaning.Therefore, " module ", " component " or " unit " can mix
Ground uses.
It should be noted that description and claims of the invention are received and the term " first " in above-mentioned attached drawing, " the
Two " etc. be to be used to distinguish similar objects, without being used to describe a particular order or precedence order.
Please refer to Fig. 1.The embodiment of the present invention provides a kind of 3D point cloud processing unit 100, comprising: 2D camera mould group 10,3D
Camera mould group 20, camera bearing support 30 and accurate carrying platform 40, in which:
The precision carrying platform 40, is located in X/Y plane, in the X/Y plane precise motion, for carrying target work
Part；
The 2D camera mould group 10, is fixed on the camera bearing support 30, shooting direction is perpendicular to the precision
Carrying platform 40, for acquiring the 2D image information for obtaining the target workpiece；
The 3D camera mould group 20, is fixed on the camera bearing support 30, with the opposite position of the 2D camera mould group 10
It sets, shooting direction is perpendicular to the accurate carrying platform 40, and the direction XY is arranged identical with the 2D camera mould group 10, is used for
Acquisition obtains the 3D point cloud of the target workpiece；
The camera bearing support 30, it is described for the 2D camera mould group 10 and the 3D camera mould group 20 to be fixed on
Accurate 40 top of carrying platform, and the relative positional relationship of fixed the 2D camera mould group 10 and the 3D camera mould group 20.Its
In, the camera bearing support 30 is fixed on the wall, alternatively, being fixed on the fixation position of production line workbench, fixed bit
It sets not shown in Fig. 1.
Further, the 2D camera mould group 10, is also used to: based on adopting, the 2D image that the acquisition of 2D camera mould group obtains is special
Sign is extracted 2D ROI (Region Of Interest, area-of-interest)；It specifically includes:
According to the inspected feature of target workpiece, 2D image is handled, extract 2D ROI, obtains it on the 2 d image
Location information MROI-2D.
Further, the 3D camera mould group 20, is also used to: by the 2D ROI backprojection to 3D point cloud space, obtaining
The 3D ROI for meeting 2D ROI must be projected, and a cloud computing is carried out to 3D ROI, 3D parameter required for obtaining；It specifically includes:
According to the transition matrix T23 of the picture position 2D to 3D point cloud X/Y plane, by 2D ROI backprojection to 3D point cloud sky
Between, range of the corresponding 3D ROI on X/Y plane is MROI-2D-3D；Wherein, the transition matrix T23 is to 2D camera mould
What group and 3D camera mould group establish after collaboration calibration；
By in 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D ROI；
Point cloud in 3D ROI is calculated, 3D parameter required for obtaining.
Preferably, the core component of the 2D camera mould group 10 uses CCD camera.
Preferably, the core component of the 3D camera mould group 20 sweeps camera using laser rays.
The embodiment of the present invention provides a kind of 3D point cloud processing unit, comprising: 2D camera mould group, 3D camera mould group, camera are held
Carrying bracket and accurate carrying platform, in which: the precision carrying platform is located in X/Y plane, in the X/Y plane precise motion,
For carrying target workpiece；The 2D camera mould group, is fixed on the camera bearing support, shooting direction is perpendicular to described
Accurate carrying platform, for acquiring the 2D image information for obtaining the target workpiece；The 3D camera mould group, is fixed on the phase
On machine bearing support, and 2D camera mould group relative position, shooting direction is perpendicular to the accurate carrying platform, and the side XY
To identical as the 2D camera mould group setting, for acquiring the 3D point cloud for obtaining the target workpiece；The camera bearing support,
For the 2D camera mould group and the 3D camera mould group to be fixed on above the accurate carrying platform, and the fixed 2D phase
The relative positional relationship of machine mould group and the 3D camera mould group.Through the embodiment of the present invention, at the image by integrating 2D and 3D
Adjustment method first does a cloud computing through 2D image zooming-out 2D ROI, then the point cloud spatial extraction 3D ROI of backprojection to 3D, thus
Under the premise of not influencing accuracy of identification, the 3D point cloud quantity calculated needed for substantially reducing improves arithmetic speed, improves operation effect
The effect of rate saves computing resource, improves 3D point cloud application real-time performance, optimizes the real-time processing of equipment in actual production
Speed is obviously improved 3D treatment efficiency.
Please refer to Fig. 2.The embodiment of the present invention provides a kind of 3D point cloud processing method, is applied to 3D point cloud processing unit, institute
Stating 3D point cloud processing unit includes: 2D camera mould group, 3D camera mould group and by the 2D camera mould group and the 3D camera mould
Group is fixed on the camera bearing support above the accurate carrying platform in X/Y plane, wherein the described method includes:
Step S1, collaboration calibration is carried out to 2D camera mould group and 3D camera mould group；
Step S2,2D camera mould group and 3D camera Mo Zu take figure respectively, obtain the 2D image and 3D point cloud of target workpiece；
Step S3, the 2D characteristics of image obtained based on 2D camera mould group extracts 2D ROI；
Step S4, by the 2D ROI backprojection to 3D point cloud space, the 3D ROI that projection meets 2D ROI is obtained；
Step S5, a cloud computing is carried out to 3D ROI, 3D parameter required for obtaining.
Further, described that collaboration calibration is carried out to 2D camera mould group and 3D camera mould group in the step S1；Include:
2D camera mould group is individually demarcated；
3D camera mould group is individually demarcated；
Collaboration calibration is carried out to 2D camera mould group and 3D camera mould group, the picture position 2D is established and turns to 3D point cloud X/Y plane
Change matrix T23.
Wherein, after the completion of collaboration calibration, the picture that the Picture Coordinate and 3D camera mould group that 2D camera mould group generates generate is sat
Mark is consistent, later can be with continuous work, without re-scaling every time.
Further, in the step S2, the 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain target workpiece
2D image and 3D point cloud；Include:
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece；
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece.
Or
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece；
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece.
Further, in the step S3, the 2D characteristics of image obtained based on 2D camera mould group extracts 2D ROI；
Include:
According to the inspected feature of target workpiece, 2D image is handled, extract 2D ROI, obtains it on the 2 d image
Location information MROI-2D.Generally, this process time-consuming is in 10~100ms rank.
Further, described by the 2D ROI backprojection to 3D point cloud space in the step S4, obtain projection symbol
Close the 3D ROI of 2D ROI；Include:
According to the transition matrix T23 of the picture position 2D to 3D point cloud X/Y plane, by 2D ROI backprojection to 3D point cloud sky
Between, range of the corresponding 3D ROI on X/Y plane is MROI-2D-3D；
By in 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D ROI.
Further, described that a cloud computing is carried out to 3D ROI in the step S5,3D parameter required for obtaining；Packet
It includes: the point cloud in 3D ROI is calculated, 3D parameter required for obtaining.
The embodiment of the present invention provides a kind of 3D point cloud processing method, is applied to a kind of 3D point cloud processing unit, comprising: to 2D
Camera mould group and 3D camera mould group carry out collaboration calibration；2D camera mould group and 3D camera Mo Zu take figure respectively, obtain target workpiece
2D image and 3D point cloud；Based on the 2D characteristics of image that 2D camera mould group obtains, 2D ROI is extracted；The 2D ROI is reversely thrown
It is mapped to 3D point cloud space, obtains the 3D ROI that projection meets 2D ROI；A cloud computing is carried out to 3D ROI, 3D required for obtaining
Parameter.Through the embodiment of the present invention, by integrating the image processing algorithm of 2D and 3D, first through 2D image zooming-out 2D ROI, then instead
A cloud computing is done to the point cloud spatial extraction 3D ROI for projecting 3D, to significantly subtract under the premise of not influencing accuracy of identification
The 3D point cloud quantity calculated needed for few improves arithmetic speed, improves the effect of operation efficiency, saves computing resource, improves 3D point
Cloud application real-time performance optimizes the real-time processing speed of equipment in actual production, is obviously improved 3D treatment efficiency.
It should be noted that above method embodiment and Installation practice belong to same design, specific implementation process is detailed
See Installation practice, and the technical characteristic in Installation practice is corresponding applicable in the method embodiment, it is no longer superfluous here
It states.
The present invention is further described for explanation and specific embodiment with reference to the accompanying drawing.
Embodiment 1:
Please refer to Fig. 3 and Fig. 4.
The embodiment of the present invention provides a kind of 3D point cloud processing unit 100.In the present embodiment, the target workpiece is with sole
For be illustrated.3D point cloud processing target are as follows: find the shoes wall displacement of sole, extract 3D trace information and be sent to subsequent mechanical
Arm completes gluing operation.
The embodiment of the present invention provides a kind of 3D point cloud processing unit 100, comprising: 2D camera mould group 10,3D camera mould group 20,
Camera bearing support 30 and accurate carrying platform 40, in which:
The precision carrying platform 40, is located in X/Y plane, in the X/Y plane precise motion, for carrying sole workpiece
50。
The camera bearing support 30, it is described for the 2D camera mould group 10 and the 3D camera mould group 20 to be fixed on
Accurate 40 top of carrying platform, and the relative positional relationship of fixed the 2D camera mould group 10 and the 3D camera mould group 20.Its
In, the camera bearing support 30 is fixed on the fixation position of production line workbench, and it is not shown in Fig. 1 to fix position.
The 2D camera mould group 10, is fixed on the camera bearing support 30, shooting direction is perpendicular to the precision
Carrying platform 40, is used for:
Acquisition obtains the 2D image information of the sole workpiece 50；
Based on the 2D characteristics of image for adopting the sole workpiece 50 that the acquisition of 2D camera mould group obtains, 2D ROI is extracted；Specifically
Include:
According to the inspected feature of the sole workpiece 50,2D image is handled, 2D ROI is extracted, obtains the sole
The location information MROI-2D of workpiece 50 on the 2 d image.
The 3D camera mould group 20, is fixed on the camera bearing support 30, with the opposite position of the 2D camera mould group 10
It sets, shooting direction is perpendicular to the accurate carrying platform 40, and identical, use is arranged with the 2D camera mould group 10 for the direction XY
In:
Acquisition obtains the 3D point cloud of the sole workpiece 50；
By the 2D ROI backprojection to 3D point cloud space, the 3D ROI that projection meets 2D ROI is obtained, and to 3D
ROI carries out a cloud computing, obtains 3D parameter required for the sole workpiece 50；It specifically includes:
According to the transition matrix T23 of the picture position 2D to 3D point cloud X/Y plane, by 2D ROI backprojection to 3D point cloud sky
Between, range of the corresponding 3D ROI on X/Y plane is MROI-2D-3D；Wherein, the transition matrix T23 is to 2D camera mould
What group and 3D camera mould group establish after collaboration calibration；
By in 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D ROI；
Point cloud in 3D ROI is calculated, (the track the 3D letter of 3D parameter required for the sole workpiece 50 is obtained
Breath), as shown in Figure 4.Black is the 50 edge track of sole workpiece, and grey is ROI range.
Embodiment 2:
Please refer to Fig. 4 and Fig. 5.
The embodiment of the present invention provides a kind of 3D point cloud processing method.In the present embodiment, the target workpiece is with sole
Example, by taking sole coating technique as an example, illustrate the processing mode of present invention method.3D point cloud processing target are as follows: find shoes
The shoes wall displacement at bottom extracts 3D trace information and is sent to the completion gluing operation of subsequent mechanical arm.
The embodiment of the present invention provides a kind of 3D point cloud processing method, is applied to 3D point cloud processing unit, at the 3D point cloud
Reason device includes: 2D camera mould group, 3D camera mould group and the 2D camera mould group and the 3D camera mould group is fixed on XY
The camera bearing support above accurate carrying platform in plane, wherein the described method includes:
Step S601,2D camera mould group and 3D camera mould group are individually demarcated respectively.
Step S602, collaboration calibration is carried out to 2D camera mould group and 3D camera mould group, establishes the picture position 2D to 3D point cloud
The transition matrix T23 of X/Y plane.
Step S603, the described 2D camera mould group carries out sole workpiece to take figure, the 2D image of the raw sole workpiece；
Step S604, the described 3D camera mould group carries out taking figure to the sole workpiece, generates the 3D point of the sole workpiece
Cloud.
Step S605, the 2D characteristics of image of the sole workpiece obtained based on 2D camera mould group extracts 2D ROI；Packet
It includes: according to the inspected feature of the sole workpiece, 2D image being handled, extract 2D ROI, obtain the sole workpiece and exist
Location information MROI-2D on 2D image.
In the present embodiment, using the area-of-interest 2D ROI of sole gluing as shoes wall, it is based on Boundary extracting algorithm, is obtained
Shoes wall marginal position L shoes wall -2D, to guarantee not lose important information in subsequent projection operation, along L shoes wall -2D to inside and outside each
A distance is expanded, MROI-2D is obtained.
Step S606, according to the transition matrix T23 of the picture position 2D to 3D point cloud X/Y plane, 2D ROI backprojection is arrived
3D point cloud space, range of the corresponding 3D ROI on X/Y plane are MROI-2D-3D；
Step S607, by 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D
ROI。
In the present embodiment, by 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and is obtained
MROI-3D=Point (Px, Py, Pz) | Point (Px, Py) ∈ MROI-2D-3D }.
Step S608, a cloud computing is carried out to 3D ROI, obtains (the track the 3D letter of 3D parameter required for the sole workpiece
Breath), as shown in Figure 4.Black is the 50 edge track of sole workpiece, and grey is ROI range.
In addition, the embodiment of the present invention also provides a kind of 3D point cloud processing system, as shown in fig. 6, the cascoded rail transmits
System 900 includes: memory 902, processor 901 and is stored in the memory 902 and can transport on the processor 901
One or more capable computer program, the memory 902 and the processor 901 are coupled in by bus system 903
Together, it is provided in an embodiment of the present invention to realize when one or more of computer programs are executed by the processor 901
A kind of following steps of 3D point cloud processing method:
Step S1, collaboration calibration is carried out to 2D camera mould group and 3D camera mould group；
Step S2,2D camera mould group and 3D camera Mo Zu take figure respectively, obtain the 2D image and 3D point cloud of target workpiece；
Step S3, the 2D characteristics of image obtained based on 2D camera mould group extracts 2D ROI；
Step S4, by the 2D ROI backprojection to 3D point cloud space, the 3D ROI that projection meets 2D ROI is obtained；
Step S5, a cloud computing is carried out to 3D ROI, 3D parameter required for obtaining.
Further, described that collaboration calibration is carried out to 2D camera mould group and 3D camera mould group in the step S1；Include:
2D camera mould group is individually demarcated；
3D camera mould group is individually demarcated；
Collaboration calibration is carried out to 2D camera mould group and 3D camera mould group, the picture position 2D is established and turns to 3D point cloud X/Y plane
Change matrix T23.
Wherein, after the completion of collaboration calibration, the picture that the Picture Coordinate and 3D camera mould group that 2D camera mould group generates generate is sat
Mark is consistent, later can be with continuous work, without re-scaling every time.
Further, in the step S2, the 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain target workpiece
2D image and 3D point cloud；Include:
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece；
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece.
Or
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece；
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece.
Further, in the step S3, the 2D characteristics of image obtained based on 2D camera mould group extracts 2D ROI；
Include:
According to the inspected feature of target workpiece, 2D image is handled, extract 2D ROI, obtains it on the 2 d image
Location information MROI-2D.Generally, this process time-consuming is in 10~100ms rank.
Further, described by the 2D ROI backprojection to 3D point cloud space in the step S4, obtain projection symbol
Close the 3D ROI of 2D ROI；Include:
According to the transition matrix T23 of the picture position 2D to 3D point cloud X/Y plane, by 2D ROI backprojection to 3D point cloud sky
Between, range of the corresponding 3D ROI on X/Y plane is MROI-2D-3D；
By in 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D ROI.
Further, described that a cloud computing is carried out to 3D ROI in the step S5,3D parameter required for obtaining；Packet
It includes: the point cloud in 3D ROI is calculated, 3D parameter required for obtaining.
The method that the embodiments of the present invention disclose can be applied in the processor 901, or by the processor
901 realize.The processor 901 may be a kind of IC chip, have signal handling capacity.During realization, on
Each step for stating method can be complete by the integrated logic circuit of the hardware in the processor 901 or the instruction of software form
At.The processor 901 can be general processor, DSP or other programmable logic device, discrete gate or transistor
Logical device, discrete hardware components etc..The processor 901 may be implemented or execute disclosed each in the embodiment of the present invention
Method, step and logic diagram.General processor can be microprocessor or any conventional processor etc..In conjunction with the present invention
The step of method disclosed in embodiment, can be embodied directly in hardware decoding processor and execute completion, or be handled with decoding
Hardware and software module combination in device execute completion.Software module can be located in storage medium, which, which is located at, deposits
The step of reservoir 902, the processor 901 reads the information in memory 902, completes preceding method in conjunction with its hardware.
It is appreciated that the memory 902 of the embodiment of the present invention can be volatile memory or nonvolatile memory,
It also may include both volatile and non-volatile memories.Wherein, nonvolatile memory can be read-only memory (ROM,
Read-Only Memory), it is programmable read only memory (PROM, Programmable Read-Only Memory), erasable
Programmable read only memory (EPROM, Erasable Read-Only Memory), electricallyerasable ROM (EEROM) (EEPROM,
Electrically Erasable Programmable Read-Only Memory), magnetic RAM (FRAM,
Ferromagnetic Random Access Memory), flash memory (Flash Memory) or other memory technologies, CD only
Read memory (CD-ROM, Compact Disk Read-Only Memory), digital versatile disc (DVD, Digital Video
) or other optical disc storages, magnetic holder, tape, disk storage or other magnetic memory apparatus Disk；Volatile memory can be at random
It accesses memory (RAM, Random Access Memory), by exemplary but be not restricted explanation, the RAM of many forms
It can use, such as static random access memory (SRAM, Static Random Access Memory), static random-access are deposited
Reservoir (SSRAM, Synchronous Static Random Access Memory), dynamic random access memory (DRAM,
Dynamic Random Access Memory), Synchronous Dynamic Random Access Memory (SDRAM, Synchronous
Dynamic Random Access Memory), double data speed synchronous dynamic RAM (DDRSDRAM,
Double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random
Access memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), synchronized links
Dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), direct rambus
Random access memory (DRRAM, Direct Rambus Random Access Memory).Description of the embodiment of the present invention is deposited
Reservoir is intended to include but is not limited to the memory of these and any other suitable type.
It should be noted that above-mentioned 3D point cloud processing system embodiment and embodiment of the method belong to same design, it is specific
Realization process is detailed in embodiment of the method, and the technical characteristic in embodiment of the method is equal in the 3D point cloud processing system embodiment
Corresponding to be applicable in, which is not described herein again.
In addition, in the exemplary embodiment, the embodiment of the present invention also provides a kind of computer storage medium, specially calculate
Machine readable storage medium storing program for executing is stored with 3D in the computer storage medium for example including the memory 902 for storing computer program
One or more program of points cloud processing method, one or more program of the 3D point cloud processing method is by processor
A kind of following steps of 3D point cloud processing method provided in an embodiment of the present invention are realized when 901 execution:
Step S1, collaboration calibration is carried out to 2D camera mould group and 3D camera mould group；
Step S2,2D camera mould group and 3D camera Mo Zu take figure respectively, obtain the 2D image and 3D point cloud of target workpiece；
Step S3, the 2D characteristics of image obtained based on 2D camera mould group extracts 2D ROI；
Step S4, by the 2D ROI backprojection to 3D point cloud space, the 3D ROI that projection meets 2D ROI is obtained；
Step S5, a cloud computing is carried out to 3D ROI, 3D parameter required for obtaining.
Further, described that collaboration calibration is carried out to 2D camera mould group and 3D camera mould group in the step S1；Include:
2D camera mould group is individually demarcated；
3D camera mould group is individually demarcated；
Collaboration calibration is carried out to 2D camera mould group and 3D camera mould group, the picture position 2D is established and turns to 3D point cloud X/Y plane
Change matrix T23.
Wherein, after the completion of collaboration calibration, the picture that the Picture Coordinate and 3D camera mould group that 2D camera mould group generates generate is sat
Mark is consistent, later can be with continuous work, without re-scaling every time.
Further, in the step S2, the 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain target workpiece
2D image and 3D point cloud；Include:
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece；
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece.
Or
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece；
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece.
Further, in the step S3, the 2D characteristics of image obtained based on 2D camera mould group extracts 2D ROI；
Include:
According to the inspected feature of target workpiece, 2D image is handled, extract 2D ROI, obtains it on the 2 d image
Location information MROI-2D.Generally, this process time-consuming is in 10~100ms rank.
Further, described by the 2D ROI backprojection to 3D point cloud space in the step S4, obtain projection symbol
Close the 3D ROI of 2D ROI；Include:
According to the transition matrix T23 of the picture position 2D to 3D point cloud X/Y plane, by 2D ROI backprojection to 3D point cloud sky
Between, range of the corresponding 3D ROI on X/Y plane is MROI-2D-3D；
By in 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D ROI.
Further, described that a cloud computing is carried out to 3D ROI in the step S5,3D parameter required for obtaining；Packet
It includes: the point cloud in 3D ROI is calculated, 3D parameter required for obtaining.
It should be noted that 3D point cloud processing method program embodiment and method on above-mentioned computer readable storage medium
Embodiment belongs to same design, and specific implementation process is detailed in embodiment of the method, and the technical characteristic in embodiment of the method is upper
It states to correspond in the embodiment of computer readable storage medium and be applicable in, which is not described herein again.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the device that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or device institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or device.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal (can be mobile phone, computer, service
Device, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
The embodiment of the present invention is described with above attached drawing, but the invention is not limited to above-mentioned specific
Embodiment, the above mentioned embodiment is only schematical, rather than restrictive, those skilled in the art
Under the inspiration of the present invention, without breaking away from the scope protected by the purposes and claims of the present invention, it can also make very much
Form, all of these belong to the protection of the present invention.
Claims (9)
1. a kind of 3D point cloud processing method is applied to 3D point cloud processing unit, which is characterized in that the 3D point cloud processing unit packet
It includes: 2D camera mould group, 3D camera mould group and the 2D camera mould group and the 3D camera mould group being fixed in X/Y plane
Camera bearing support above accurate carrying platform, wherein the described method includes:
Collaboration calibration is carried out to 2D camera mould group and 3D camera mould group；
The 2D camera mould group and 3D camera Mo Zu take figure respectively, obtain the 2D image and 3D point cloud of target workpiece；
Based on the 2D characteristics of image that 2D camera mould group obtains, 2D ROI is extracted；
By the 2D ROI backprojection to 3D point cloud space, the 3D ROI that projection meets 2D ROI is obtained；
A cloud computing is carried out to the 3D ROI, 3D parameter required for obtaining.
2. the method according to claim 1, wherein described cooperate with 2D camera mould group and 3D camera mould group
Calibration；Include:
The 2D camera mould group and 3D camera mould group are individually demarcated respectively；
Collaboration calibration is carried out to 2D camera mould group and 3D camera mould group, establish the picture position 2D to 3D point cloud X/Y plane conversion square
Battle array T23.
3. according to the method described in claim 2, it is characterized in that, the 2D camera mould group and 3D camera Mo Zu take figure respectively,
Obtain the 2D image and 3D point cloud of target workpiece；Include:
The 2D camera mould group carries out target workpiece to take figure, generates the 2D image of target workpiece；
The 3D camera mould group carries out target workpiece to take figure, generates the 3D point cloud of target workpiece.
4. according to the method described in claim 3, it is characterized in that, it is described based on 2D camera mould group obtain 2D characteristics of image,
Extract 2D ROI；Include:
According to the inspected feature of target workpiece, 2D image is handled, 2D ROI is extracted, obtains its position on the 2 d image
Information MROI-2D.
5. according to the method described in claim 4, it is characterized in that, described by the 2D ROI backprojection to 3D point cloud sky
Between, obtain the 3D ROI that projection meets 2D ROI；Include:
According to the transition matrix T23, by 2D ROI backprojection to 3D point cloud space, corresponding 3D ROI is on X/Y plane
Range is MROI-2D-3D；
By in 3D point cloud space, the point that XY coordinate value is fallen in MROI-2D-3D is extracted, and obtains 3D ROI.
6. according to the method described in claim 5, it is characterized in that, described carry out a cloud computing to 3D ROI, required for acquisition
3D parameter；It include: to calculate the point cloud in 3D ROI, 3D parameter required for obtaining.
7. a kind of 3D point cloud processing unit, which is characterized in that be applied to a kind of 3D point cloud as claimed in any one of claims 1 to 6
Processing method, the 3D point cloud processing unit, comprising: 2D camera mould group, 3D camera mould group, camera bearing support and accurate carrying
Platform, in which:
The precision carrying platform, is located in X/Y plane, in the X/Y plane precise motion, for carrying target workpiece；
The 2D camera mould group, is fixed on the camera bearing support, shooting direction perpendicular to the accurate carrying platform,
For acquiring the image information for obtaining the target workpiece；
The 3D camera mould group, is fixed on the camera bearing support, with 2D camera mould group relative position, shooting side
To perpendicular to the accurate carrying platform, and the direction XY is identical with the 2D camera mould group setting, for the acquisition acquisition mesh
Mark the 3D point cloud of workpiece；
The camera bearing support is flat for the 2D camera mould group and the 3D camera mould group to be fixed on the accurate carrying
Above platform, and the relative positional relationship of fixed the 2D camera mould group and the 3D camera mould group.
8. a kind of 3D point cloud processing system characterized by comprising memory, processor and be stored on the memory simultaneously
The computer program that can be run on the processor is realized when the computer program is executed by the processor as right is wanted
Described in asking any one of 1 to 6 the step of a kind of 3D point cloud processing method.
9. a kind of computer readable storage medium, which is characterized in that stored at 3D point cloud on the computer readable storage medium
Such as any one of claims 1 to 6 is realized when the program of the program of reason method, the 3D point cloud processing method is executed by processor
A kind of the step of 3D point cloud processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910282317.1A CN110047147A (en) | 2019-04-09 | 2019-04-09 | A kind of 3D point cloud processing method, device, system and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910282317.1A CN110047147A (en) | 2019-04-09 | 2019-04-09 | A kind of 3D point cloud processing method, device, system and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110047147A true CN110047147A (en) | 2019-07-23 |
Family
ID=67276488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910282317.1A Pending CN110047147A (en) | 2019-04-09 | 2019-04-09 | A kind of 3D point cloud processing method, device, system and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110047147A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113021333A (en) * | 2019-12-25 | 2021-06-25 | 沈阳新松机器人自动化股份有限公司 | Object grabbing method and system and terminal equipment |
| CN114463165A (en) * | 2021-09-28 | 2022-05-10 | 西安大医集团股份有限公司 | Method and device for determining VOI (volume of interest) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011139854A (en) * | 2010-01-08 | 2011-07-21 | Toshiba Corp | Ultrasonic diagnosis apparatus, medical image processing apparatus, and medical image processing program |
| CN104574425A (en) * | 2015-02-03 | 2015-04-29 | 中国人民解放军国防科学技术大学 | Calibration and linkage method for primary camera system and secondary camera system on basis of rotary model |
| CN108151671A (en) * | 2016-12-05 | 2018-06-12 | 杭州先临三维科技股份有限公司 | A kind of 3 D digital imaging sensor, 3 D scanning system and its scan method |
| CN108932475A (en) * | 2018-05-31 | 2018-12-04 | 中国科学院西安光学精密机械研究所 | Three-dimensional target identification system and method based on laser radar and monocular vision |
| CN109410264A (en) * | 2018-09-29 | 2019-03-01 | 大连理工大学 | A kind of front vehicles distance measurement method based on laser point cloud and image co-registration |
-
2019
- 2019-04-09 CN CN201910282317.1A patent/CN110047147A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011139854A (en) * | 2010-01-08 | 2011-07-21 | Toshiba Corp | Ultrasonic diagnosis apparatus, medical image processing apparatus, and medical image processing program |
| CN104574425A (en) * | 2015-02-03 | 2015-04-29 | 中国人民解放军国防科学技术大学 | Calibration and linkage method for primary camera system and secondary camera system on basis of rotary model |
| CN108151671A (en) * | 2016-12-05 | 2018-06-12 | 杭州先临三维科技股份有限公司 | A kind of 3 D digital imaging sensor, 3 D scanning system and its scan method |
| CN108932475A (en) * | 2018-05-31 | 2018-12-04 | 中国科学院西安光学精密机械研究所 | Three-dimensional target identification system and method based on laser radar and monocular vision |
| CN109410264A (en) * | 2018-09-29 | 2019-03-01 | 大连理工大学 | A kind of front vehicles distance measurement method based on laser point cloud and image co-registration |
Non-Patent Citations (1)
| Title |
|---|
| 周春艳等: "数字图像辅助激光点云特征提取", 《软件时空》, vol. 27, no. 10, pages 140 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113021333A (en) * | 2019-12-25 | 2021-06-25 | 沈阳新松机器人自动化股份有限公司 | Object grabbing method and system and terminal equipment |
| CN114463165A (en) * | 2021-09-28 | 2022-05-10 | 西安大医集团股份有限公司 | Method and device for determining VOI (volume of interest) |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8818131B2 (en) | Methods and apparatus for facial feature replacement | |
| KR101532864B1 (en) | Planar mapping and tracking for mobile devices | |
| KR101585521B1 (en) | Scene structure-based self-pose estimation | |
| US9635251B2 (en) | Visual tracking using panoramas on mobile devices | |
| CN112116639B (en) | Image registration method and device, electronic equipment and storage medium | |
| CN106033621B (en) | A kind of method and device of three-dimensional modeling | |
| CN106355550B (en) | Image stitching system and image stitching method | |
| CN108288292A (en) | A kind of three-dimensional rebuilding method, device and equipment | |
| CN112348890B (en) | Space positioning method, device and computer readable storage medium | |
| CN112102380A (en) | Registration method and related device for infrared image and visible light image | |
| CN114283079B (en) | A method and device for photographing and correcting based on a chart | |
| CN111383264B (en) | Positioning method, positioning device, terminal and computer storage medium | |
| CN110047147A (en) | A kind of 3D point cloud processing method, device, system and computer storage medium | |
| CN114022542A (en) | A method of making 3D database based on 3D reconstruction | |
| CN115810052A (en) | Camera calibration method, device, electronic equipment and storage medium | |
| Jin et al. | Parallax tolerant light field stitching for hand-held plenoptic cameras | |
| US9449426B2 (en) | Method and apparatus for centering swivel views | |
| Kratochvil et al. | Image‐based 3D reconstruction using helical nanobelts for localized rotations | |
| JP2020135092A (en) | Learning data generator, method and program | |
| CN113807192B (en) | A multi-target recognition calibration method for augmented reality | |
| CN119181046B (en) | Logistics product positioning data management method and system based on cloud computing | |
| CN117506919B (en) | Hand-eye calibration method and device, terminal equipment and storage medium | |
| CN113744298B (en) | Data processing method, device and storage medium | |
| CN111950395B (en) | A vehicle identification method, device and computer storage medium | |
| Wang et al. | Three-dimensional reconstruction coordinate error induced by asynchronous cameras for moving objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190723 |
|
| RJ01 | Rejection of invention patent application after publication |