CN110502948B - Restoration method and device for folding two-dimensional code image and code scanning equipment - Google Patents

Abstract

The invention discloses a method for restoring a folded two-dimensional code image, a restoring device and a code scanning device. The restoration method includes: an image preprocessing step, preprocessing a folded two-dimensional code image to obtain a preprocessed image containing two-dimensional code image information; a corner detection step, performing corner detection on the preprocessed image , to obtain a plurality of correction points of the folded two-dimensional code image, wherein the plurality of correction points at least include the top corner point of the folded two-dimensional code image and the folding corner point located on the folding line of the folded two-dimensional code image ; Image restoration step, using the corner point and the top corner point to perform perspective restoration on the folded two-dimensional code image, so as to obtain the first part image and the second part of the folded two-dimensional code image in two intersection planes Image; an image splicing step, splicing the first part of the image and the second part of the image to obtain a plane two-dimensional code image. The invention can realize extracting information from folded two-dimensional code images.

Description

Method for restoring folded two-dimensional code image, restoring device and code scanning device

technical field

The present invention is related to the application of two-dimensional code images, in particular to a method for restoring a folded two-dimensional code image, a restoring device and a code scanning device with the related restoring device.

Background technique

Image QR code applications, such as QR Code, have been widely used in various fields. The image of the two-dimensional code used to carry information is usually square and often displayed on a plane (such as a label pasted on a commodity packaging box). After the two-dimensional code image is captured by the camera, the lens and sensor of the camera are usually not parallel to the plane two-dimensional code, and the capturing angle can be any angle under the premise that the information carried by the two-dimensional code image can be restored and extracted. QR code images usually cause different degrees of perspective distortion.

Perspective distortion will cause the ingested QR code image to be different from the original QR code image in shape, size, and position, and the information carried by the ingested QR code image cannot be captured before perspective restoration. Extraction, so perspective restoration is a necessary part in the application of image QR codes, and the method of its realization has also been studied a lot. The most commonly used method is to use the four corner points of the two-dimensional code image to add the finder image as the position correction point. After the four corner points are extracted by other image processing techniques, the captured corner points can be used The coordinates and the corresponding original coordinates are used to calculate the perspective distortion experienced by the captured image and restore it. For example, the correction point used by the widely used ZXing open source barcode software library is the image-finding image of the QR code itself. In addition to using the image-finding image as a correction point, the structure of the two-dimensional code image itself can also be used to correct perspective distortion. For example, in the Chinese patent application CN101908144A, the position of the module in the captured two-dimensional code image can be obtained Multiple correction points that can be used to calculate perspective distortion parameters.

Based on the perspective restoration of the plane QR code image, there is also a work on the perspective distortion to enhance the perspective restoration after it is ingested in a certain non-planar situation, to deal with when the plane QR code is pasted The surface of the label has a slight curvature, or the surface of the QR code image label itself has irregular shape changes other than perspective distortion. For example, Chinese patent application CN201510432188 uses a matching template to correct perspective distortion and surface correction by using multiple image-finding images.

However, at present, these related two-dimensional code perspective restoration methods generally only restore the two-dimensional code image displayed on the plane, and have certain curvature and other distortions, without considering the folding deformation of the two-dimensional code image, such as in the middle position The deformation caused by being folded and glued to a product package consisting of two flat surfaces at an angle. This kind of folding deformation causes the original plane two-dimensional code image to become a three-dimensional two-dimensional code folded at the intersection of two planes (such as two adjacent faces of a cube), so the traditional two-dimensional code image perspective restoration The method cannot be used for QR code images that are folded and pasted from the middle. Therefore, this type of packaging currently cannot place the QR code image at the intersection of the two planes. However, this type of commodity packaging is more common, ranging from cube-shaped packaging to all packaging where two intersecting planes exist.

Therefore, there is an urgent need for a method for restoring a folded two-dimensional code image, which can extract information from the folded two-dimensional code image.

Contents of the invention

In view of this, an object of the present invention is to provide a method for restoring a folded two-dimensional code image, its device, and a code scanning device, so that information can be extracted from the image two-dimensional code pasted at the plane intersection of the product package. possible.

In order to achieve the above object, the present invention provides a method for restoring a folded two-dimensional code image, which is characterized in that it includes:

The image preprocessing step is to preprocess a folded two-dimensional code image to obtain a preprocessed image containing two-dimensional code image information;

The corner point detection step is to perform corner point detection on the preprocessed image to obtain a plurality of correction points of the folded two-dimensional code image, wherein the plurality of correction points at least include the top corner point of the folded two-dimensional code image and A folding corner located on a folding line of the folding two-dimensional code image;

The image restoration step is to use the folded corners and the top corners to perform perspective restoration on the folded two-dimensional code image, so as to obtain the first part image and the second part of the folded two-dimensional code image in two intersection planes image;

The image splicing step is to splice the first part of the image and the second part of the image to obtain a plane two-dimensional code image.

In one embodiment of the present invention, the image preprocessing step includes:

Performing morphological processing on the folded two-dimensional code image, by distinguishing different objects in the folded two-dimensional code image, extracting the two-dimensional code image information, wherein the two-dimensional code image information includes the position information of the two-dimensional code image .

In one embodiment of the present invention, before performing morphological processing on the folded two-dimensional code image, it also includes:

performing grayscale conversion processing on the folded two-dimensional code image to obtain a grayscale image;

Binarize the gray value image to obtain a black and white binary image.

In one embodiment of the present invention, when performing morphological processing on the folded two-dimensional code image, it includes:

performing a closing operation on the black-and-white binary image by using structural elements, so that the boundaries of the two-dimensional code images in the black-and-white binary image become continuous;

The black and white binary image is filled to distinguish the two-dimensional code image from other objects, and the position information of the two-dimensional code image in the black and white binary image is obtained.

In one embodiment of the present invention, the corner detection step includes:

Use the filled black and white binary image as a label image, and perform edge detection on it to obtain multiple edges of the two-dimensional code image, and obtain an edge label image;

performing line detection on the edge label image to extract edge information of the multiple edges of the two-dimensional code image;

Corner detection is performed by using the edge information of the plurality of edges to obtain a plurality of top corners and folding corners on the folding line of the two-dimensional code image.

In an embodiment of the present invention, in the image restoration step, the obtained restored first partial image and the second partial image are square images with an aspect ratio set to 1 after correction.

In one embodiment of the present invention, the image stitching step includes:

Calculate the number of modules of the two-dimensional code image in the horizontal direction and the vertical direction respectively by the module number estimation method;

According to the width of the first part image and the second part image of the restored two-dimensional code image, and the calculated number of modules in the horizontal direction and the vertical direction, the second part of the restored two-dimensional code image is The part of the image and the second part of the image are adjusted from the square image to their original aspect ratio to obtain the corrected first part of the image and the corrected second part of the image;

splicing the corrected first part image and the corrected second part image to obtain the planar two-dimensional code image.

In an embodiment of the present invention, the method for estimating the number of modules includes:

In the black-and-white binary image, the scanning position of the horizontal positioning figure is determined by the two top corner points obtained in the horizontal direction, and the horizontal position of the two-dimensional code image is determined according to the scanning result of the horizontal positioning figure. number of modules on

In the black-and-white binary image, the scanning position of the vertical positioning figure is determined by the top corner and folding corner obtained in the vertical direction, and the two-dimensional code image is determined according to the scanning result of the vertical positioning figure. The number of modules in the vertical direction.

In one embodiment of the present invention, when the detection of the corner point detection step fails or the image restoration step fails, it also includes:

The closing operation step is repeated by increasing the size of the structuring element.

In one embodiment of the present invention, before performing gray value conversion processing on the folded two-dimensional code image, it also includes:

A captured image is obtained by an image capturing device, and the captured image is cut to obtain the folded two-dimensional code image with a predetermined size.

In order to achieve the above object, the present invention further provides a device for restoring a folded two-dimensional code image, which is characterized in that it includes:

An image preprocessing unit, configured to preprocess a folded two-dimensional code image to obtain a preprocessed image containing two-dimensional code image information;

A corner detection unit, configured to perform corner detection on the preprocessed image to obtain a plurality of correction points of the folded two-dimensional code image, wherein the plurality of correction points include at least the top corner of the folded two-dimensional code image point and the folded corners on the folded line of the folded two-dimensional code image;

An image restoring unit, configured to use the folded corners and the top corners to perform perspective restoration on the folded two-dimensional code image, so as to obtain the first part image and the second part of the folded two-dimensional code image in the two intersection planes two-part image;

The image splicing unit is used to splice the first partial image and the second partial image to obtain a plane two-dimensional code image.

In another embodiment of the present invention, the image preprocessing unit includes:

The morphological processing module is used to perform morphological processing on the folded two-dimensional code image, and extract the two-dimensional code image information by distinguishing different objects in the folded two-dimensional code image, wherein the two-dimensional code image information contains two The location information of the QR code image.

In another embodiment of the present invention, the image preprocessing unit also includes:

A grayscale value conversion processing module, configured to perform grayscale value conversion processing on the folded two-dimensional code image to obtain a grayscale value image; and

The binarization processing module is used to perform binarization processing on the gray value image to obtain a black and white binary image.

In another embodiment of the present invention, when the morphological processing module performs morphological processing on the folded two-dimensional code image, it uses structural elements to perform a closed operation on the black and white binary image, so that the binary in the black and white binary image The boundary of the two-dimensional code image becomes continuous; and the black and white binary image is filled to distinguish the two-dimensional code image from other objects, and the position information of the two-dimensional code image in the black and white binary image is obtained.

In another embodiment of the present invention, the corner detection unit includes:

An edge detection module, configured to use the filled black and white binary image as a label image, and perform edge detection on it to obtain multiple edges of the two-dimensional code image, and obtain an edge label image;

A line detection module, configured to perform line detection on the edge label image, to extract edge information of the plurality of edges of the two-dimensional code image;

The corner detection module is configured to use the edge information of the plurality of edges to perform corner detection, so as to obtain a plurality of top corners of the two-dimensional code image and folding corners located on the folding line.

In another embodiment of the present invention, the restored first partial image and the second partial image obtained by the image restoration unit are square images with an aspect ratio set to 1 after correction.

In another embodiment of the present invention, the image stitching unit includes:

The module quantity estimation module is used to calculate the module quantity of the two-dimensional code image in the horizontal direction and the vertical direction respectively by the module quantity estimation method;

The image size adjustment module is used to adjust the restored two-dimensional code image according to the width of the first part of the image and the width of the second part of the image, and the calculated number of modules in the horizontal and vertical directions. The first part image and the second part image of the two-dimensional code image are adjusted to their original aspect ratio by the square image, so as to obtain the corrected first part image and the corrected second part image;

The image splicing module is used to splice the corrected first part of the image and the corrected second part of the image to obtain the planar two-dimensional code image.

In another embodiment of the present invention, the module number estimation module is:

In the black-and-white binary image, the scanning position of the horizontal positioning figure is determined by the two top corner points obtained in the horizontal direction, and the horizontal position of the two-dimensional code image is determined according to the scanning result of the horizontal positioning figure. number of modules on

In the black-and-white binary image, the scanning position of the vertical positioning figure is determined by the top corner and folding corner obtained in the vertical direction, and the two-dimensional code image is determined according to the scanning result of the vertical positioning figure. The number of modules in the vertical direction.

In another embodiment of the present invention, when the corner detection step fails or the image restoration step fails, the morphological processing module further performs the closing operation step by increasing the size of the structural elements.

In another embodiment of the present invention, the image preprocessing unit also includes:

The cutting module is used to obtain a captured image through an image capturing device, and perform cutting processing on the captured image to obtain the folded two-dimensional code image with a predetermined size.

In order to achieve the above purpose, the present invention further provides a code scanning device for folding two-dimensional code images, which is characterized in that it includes:

A camera unit for capturing a folded two-dimensional code image;

A device for restoring the folded two-dimensional code image as described above, used to restore the folded two-dimensional code image to obtain a plane two-dimensional code image;

A two-dimensional code decoder is used to decode the restored flat two-dimensional code image.

Aiming at the perspective distortion of folding deformation, the present invention designs a related restoration method and its device and related code scanning equipment without changing the structure of the two-dimensional code of the image itself, so that the planes located on the commodity packaging meet It is possible to extract information from image QR codes pasted everywhere.

The above-mentioned description will be described in detail in the following embodiments, and a further explanation will be provided for the technical solution of the present invention.

Description of drawings

In order to make the above and other objects, features, advantages and embodiments of the present invention more comprehensible, the accompanying drawings are described as follows:

Fig. 1 is a flowchart of a method for restoring a folded two-dimensional code image in an embodiment of the present invention;

Fig. 2 is a schematic diagram of a folded two-dimensional code image captured by a camera in an embodiment of the present invention;

Fig. 3 is a schematic diagram of a folded two-dimensional code image after cutting and gray value conversion in an embodiment of the present invention;

FIG. 4A is a schematic diagram showing the relationship between image blocks and captured images during binarization processing according to an embodiment of the present invention;

Fig. 4B is a black and white binary image obtained after binarization processing in an embodiment of the present invention;

5 is a schematic diagram of a structural element E _c used in the process of morphological processing in an embodiment of the present invention;

Fig. 6 is a schematic diagram of the closed operation result image _Ic obtained after the closed operation is performed during the morphological processing in an embodiment of the present invention;

Fig. 7 is a schematic diagram of the filling result image If obtained after filling during the _{morphological} processing in an embodiment of the present invention;

8 is a schematic diagram of an edge label image I _e obtained after edge detection in an embodiment of the present invention;

9 is a schematic diagram of the relationship between Hough domain coordinates (θ, ρ) and pixel coordinates (x, y) during line detection in an embodiment of the present invention;

Figure 10A and Figure 10B are schematic diagrams of the Hough transform results in an embodiment of the present invention, wherein Figure 10A shows the representation of the upper and lower four top corners of the folded two-dimensional code image in the Hough domain, and Figure 10B shows the representation of the folded two-dimensional code image The representation of the two folding corners on the middle folding line of the two-dimensional code image in the Hough domain;

Fig. 11 is a schematic diagram of the results after the line detection in an embodiment of the present invention;

Fig. 12 is a schematic diagram of the results after corner detection in an embodiment of the present invention;

Fig. 13A and Fig. 13B are schematic diagrams of perspective restoration results obtained after the image restoration step in an embodiment of the present invention, wherein Fig. 13A shows the first part of the image (ie, the upper image) of the folded two-dimensional code image after restoration, and Fig. 13B shows the second part of the image (i.e. the lower image) of the folded two-dimensional code image after reduction;

Fig. 14 is a schematic diagram of calculating the number of modules in the horizontal direction by the method of estimating the number of modules in an embodiment of the present invention, which shows the levels determined from the extracted two calibration points at the lower end in the black-and-white binary image Locate the scanning position of the graphic;

Fig. 15 is a schematic diagram of the scanning result of the horizontal positioning pattern in Fig. 14;

FIG. 16A and FIG. 16B are respectively schematic diagrams of the corrected first part of the image and the corrected second part of the image obtained after image size adjustment in an embodiment of the present invention;

Fig. 17 is a schematic diagram of a plane two-dimensional code image obtained after the image stitching step in an embodiment of the present invention;

Fig. 18 is a schematic structural diagram of a device for restoring a folded two-dimensional code image according to the present invention;

FIG. 19 is a schematic structural diagram of a code scanning device for folding two-dimensional code images according to the present invention.

Detailed ways

In order to make the description of the present invention more detailed and complete, reference may be made to the attached drawings and various embodiments described below, and the same numbers in the drawings represent the same or similar components. On the other hand, well-known components and steps have not been described in the embodiments in order not to limit the invention unnecessarily. In addition, for the sake of simplifying the drawings, some known and conventional structures and elements will be shown in a simple and schematic manner in the drawings.

The method for restoring a folded two-dimensional code image according to the present invention will be described in detail in a preferred embodiment below with reference to FIGS. 1 to 17 .

It needs to be explained in advance that the restoration method disclosed in this preferred embodiment assumes that the original two-dimensional code image is quadrilateral (not square) and the folds are parallel to the upper and lower sides, and the bit information carried by each of them is composed of a It is composed of square modules, and there are alternating black and white modules as positioning graphics on the side passing through the intersection of the planes and one of the upper and lower sides. The size of the black and white modules in the positioning graphics is the same as that of the data module. A module row and module column are positioned throughout the module section of the data. However, it can be understood that the above is only the assumption of the two-dimensional code in this preferred embodiment, but the methods, devices and applications involved in the present invention do not limit the types of two-dimensional codes and the like.

Moreover, after folding with an unknown angle, the original planar two-dimensional code image is composed of two different intersecting planes. The restored planes are spliced to restore the original two-dimensional code image. The invention does not assume a particular folding angle, but assumes that it is large enough to distinguish between the upper and lower planes without causing either of the two planes to be obscured by the other or out of focus.

As shown in FIG. 1 , it shows the flow of a method for restoring a folded two-dimensional code image in a preferred embodiment of the present invention.

The restoration method is divided into four main steps: an image preprocessing step, a corner point detection step, an image restoration step, and an image stitching step.

Image preprocessing steps

The image preprocessing step is mainly to preprocess a folded two-dimensional code image to obtain a preprocessed image containing two-dimensional code image information. In this preferred embodiment, the image preprocessing step may include multiple processing steps such as cutting processing, gray value conversion processing, binarization processing, and morphological processing. However, it can be understood that, in other embodiments, only one or more of these processing steps may be included, and these are not intended to limit the present invention.

Carry out cutting processing, gray value conversion processing and binarization processing on the captured image

In this preferred embodiment, it is assumed that an image capture device, such as a camera unit of a code scanning device, captures the folded two-dimensional code image after a fold with an unknown folding angle, and the folding line and the two-dimensional code image Two parallel sides among the four sides are parallel, and the image of the folded two-dimensional code captured by the camera unit is shown in FIG. 2 .

First of all, when obtaining the captured folded two-dimensional code image, in order to reduce the calculation load of the subsequent image processing, the captured image can be cut to reduce the number of processed pixels and other irrelevant image parts, so as to obtain a The folded two-dimensional code image of a predetermined size, for example, can be cut to obtain a folded two-dimensional code image of 675×675 pixels. In the actual operation process, by designing the code scanning device and with the assistance of the user, the camera unit of the code scanning device can make the two-dimensional code roughly in the center when capturing the image, so that a predetermined size (such as 675× 675 pixels) of the folded QR code image, so as to reduce the range of images that need to be processed later and increase the overall calculation speed. The folded two-dimensional code image of the predetermined size can be used as an input for subsequent steps.

Since no color information is involved, the acquired folded two-dimensional code image can be converted into a gray value image I _g using a known conversion formula.

I _g =0.299×R+0.587×G+0.114×B

Where (R, G, B) are the three color values of red, green and blue of the captured folded QR code image. After the cutting process and the gray value conversion process, the gray value image I _g to be processed subsequently is shown in FIG. 3 .

After the gray value image I _g is obtained, it is necessary to distinguish the two-dimensional code image part from other irrelevant image parts. In this step, the gray value image I _g is further transformed into a binary image, that is, a black and white binary image I _b .

Considering the uneven reflected light and the different contrast of the pixel values in the two-dimensional code image content, a simple adaptive threshold setting method is as follows from the ZXing software library: First, decompose the captured image into several 8×8 pixel image blocks. If its size is not a multiple of 8, the image block size of the boundary can be smaller than 8×8 pixels. Use blk(i,j) to represent the (i,j)th image block, in each image block, calculate:

The relationship between the image block and the captured image is shown in Figure 4A, wherein, assuming that the captured image has a size of 24×24 pixels, it can be divided into 9 image blocks, for example, including image blocks blk(1,1), blk (1,2), blk(1,3), blk(2,1), ..., blk(3,3), etc., the size of each image block is 8×8 pixels.

Then, take (i, j) as the center, and take the average value of p(i, j) in the surrounding 5×5 image blocks as the threshold.

If the surrounding image blocks exceed the boundary of the image, the range is adjusted to the nearest 5×5 image blocks to (i, j). Finally, in the (i,j)th image block, compare the gray value image with the corresponding threshold to obtain the black and white binary image I _b :

(k,l)位于图像块blk(i,j)中

(k,l) is located in the image block blk(i,j)

The obtained black and white binary image I _b is shown in Fig. 4B.

Morphological processing

This step is to perform morphological processing on the folded two-dimensional code image, by distinguishing different objects in the folded two-dimensional code image, and extract the two-dimensional code image information therein, wherein the two-dimensional code image information includes the two-dimensional code image location information.

In this step, the obtained black-and-white binary image I _b is used to distinguish different objects in the captured image, extract the two-dimensional code image information therein, and remove other unnecessary image parts. Here, it is assumed that the two-dimensional code image is the most important object in the captured image.

First, perform a closing operation on the black and white binary image I _b . This step is to use the standard closing operation with the structural element as E _c to make the boundary of the two-dimensional code image in the black and white binary image continuous. Wherein, this closing operation step is to bridge the breaks in the periphery and boundary of objects (including two-dimensional code images and other possible objects) in the black and white binary image, while maintaining the overall position and position of the object in the black and white binary image. shape:

Here, since it is impossible to predict in advance the capturing environment when the camera captures the two-dimensional code image, the number of surrounding objects that interfere with the detection of the two-dimensional code image, and the proportion of the two-dimensional code itself in the entire image, the structural element E _c can be set first is a circular sliding window with a radius of 7, as shown in Figure 5.

表示图像形态的膨胀操作：in addition,

Represents a dilation operation for image morphology:

It will cause the boundary of the object to expand outward according to the structure of the structural element E _c , and ∪ can be used as an "or (or)" operation.

代表图像形态的腐蚀操作：

Corrosion operations representing image morphologies:

It will cause the boundary of the object to shrink inward according to the structural element _Ec , that is, only when the part of the shape at the boundary of the image coincides with _Ec will be retained, ∩ can be used as an "and" operation.

After the closing operation, the boundary of the part of the two-dimensional code image in the black and white binary image will become continuous, as shown in FIG. 6 , which shows the result image I _c of the closing operation obtained after the closing operation. Compared with FIG. 4B , it can be seen that the boundary of the two-dimensional code image in the middle in FIG. 4B has been connected as a whole.

In the present invention, when the subsequent steps fail, for example, when the corner detection step fails or the image restoration step fails, the closing operation step can be re-performed by increasing the size of the structural element, and the subsequent steps can be tried again. For example, the closing operation step can be performed again by increasing the radius of the window to fill in a larger range of different objects on the image. In this way, the present invention can adjust the structural elements in the closing operation through the results of the corner detection step and the image restoration step to control the entire process.

The next step is to fill the image to distinguish the QR code image from other objects, and obtain the position of the QR code image in the captured image. Since after the closing operation, the boundaries of the same objects in the closing operation result image I _c have been connected as a whole, and different objects are not connected, then different objects can be filled separately, and the largest area (white pixel points) can be obtained most) objects as the position of the QR code image. The filled result image If after _filling is shown in FIG. 7 .

Corner detection step

A corner point detection step, which mainly performs corner point detection on the preprocessed image to obtain a plurality of correction points of the folded two-dimensional code image, wherein the plurality of correction points include at least the top corner point of the folded two-dimensional code image and the folded corners located on the folded line of the folded two-dimensional code image. In this preferred embodiment, the corner detection step may include multiple detection steps such as edge detection, line detection and corner detection.

edge detection

It can be seen from the captured image of the folded QR code that there are 6 corners and 6 sides in the folded QR code. Therefore, by performing edge detection and line detection on the captured image, 6 sides can be determined, and 6 corner points can be determined according to their intersection points, for example, including the 4 top corner points of the folded two-dimensional code image and Two folding corners located on the folding line of the folded QR code image.

In this step, the filled black-and-white binary image can be used, for example, the filled result image _If as the label image for edge detection. Since the edge of the white part in Figure 7 is very obvious, and the edge to be extracted is also very specific, the edge detection required here is not difficult, so different edge detection methods can be used, such as Prewitt or Sobel. For example, this preferred embodiment can use the Sobel Operator to extract the edges in the black and white binary image, and its sliding windows in the horizontal and vertical directions are respectively:

Then, the edge with a larger magnitude can be extracted as the edge of the two-dimensional code image. The final edge label image I _e obtained is:

where * represents convolution, and the resulting edge label image I _e is shown in Figure 8.

line detection

After the edge label image I _e is obtained, the edge information needs to be extracted for the next step of processing. Here, the commonly used Hough transform can be used for line detection. For each white pixel point (x, y) of the label image on the edge of the QR code, after the Hough transform, it becomes (θ, ρ) through the following relationship:

x cosθ+y sinθ=ρ

And add 1 to (θ,ρ) in the Hough domain. After the Hough transform is performed on the edge label image _Ie , the Hough transform results shown in Figs. 10A and 10B are obtained. The coordinates (x, y) here are the coordinates in the captured two-dimensional code image, and its origin is the upper left corner of the image, as shown in Figure 9, which shows the relationship between the Hough domain coordinates (θ, ρ) and the pixel coordinates (x,y) relationship. Each pixel on the straight line L will add 1 to the same (θ, ρ), and finally accumulate to the local maximum value of the Hough domain.

Figure 10A and Figure 10B show the results of the Hough transform respectively, Figure 10A shows the representation of the upper and lower four top corners of the folded two-dimensional code image in the Hough domain, Figure 10B shows the fold line in the middle of the folded two-dimensional code image A representation of the 2 folded corners in the Hough domain. In FIG. 10A and FIG. 10B , 4 straight lines are obtained in the vertical direction, and 2 straight lines are obtained in the horizontal direction, which are respectively marked by white squares as local maxima in the Hough domain.

Although the edges obtained in the edge detection step are line segments, the straight lines where the six detected line segments are located are used here, which can be represented by homogeneous coordinates. In this way, the information of the edge straight line can be obtained in the Hough domain, including its starting point s=(x _s ,y _s ,1) and ending point e=(x _e ,y _e ,1). In the homogeneous coordinate system, l=[a,b,c] can be used to represent the straight line, then the points on the straight line satisfy:

ax+by+c=0

Among them, the straight line l can be obtained by the cross product of two points on the straight line:

Here s=(x _s ,y _s ,1) and e=(x _e ,y _e ,1) can be calculated from the point (θ,ρ) of the straight line in the Hough domain. The obtained 4 straight lines in the vertical direction can be represented by l _i , i=1,...,4, as shown in the straight line li in Figure 11, and the 2 straight lines in the horizontal direction can be represented by l _j , _j =1, 2, as shown in Figure 11 The straight line l _j in .

Corner detection

Before performing image restoration, it is necessary to obtain the four top corners of the captured folded QR code image and the two fold corners at the fold as input for correction points and image restoration. After obtaining the straight lines l _i and l _j of the two-dimensional code image boundary, the homogeneous coordinates of the intersection points of the straight lines can be obtained by cross product again:

Wherein, the four top corners of the two-dimensional code image are the intersection points of the straight line l _j in the horizontal direction and the straight line l _i in the vertical direction.

Two folding corner points at the folding place of the two-dimensional code image are intersection points between straight lines l _i in different vertical directions.

为两条不同的垂直线。并且，在这些交点中，只需取在摄取到的图像范围中的交点，作为需要的校正点。另外，校正点的坐标需要转换为二维平面坐标c。in,

for two different vertical lines. And, among these intersection points, only the intersection points in the captured image range are taken as required correction points. In addition, the coordinates of the calibration points need to be transformed into two-dimensional plane coordinates c.

As shown in FIG. 12, 6 correction points are extracted from the straight line of the boundary of the two-dimensional code image. Among them, P1-P4 are the four top corners of the two-dimensional code image, and P5-P6 are the two folding corners of the folding place.

Image Restoration Steps

The image restoration step mainly uses the folded corners and the top corners to perform perspective restoration on the folded two-dimensional code image, so as to obtain the first part of the folded two-dimensional code image in the two intersection planes and The second part of the image.

Mathematical Expression of Perspective Transformation

In the present invention, it is assumed that the camera model is a pinhole camera model. Although other, more accurate camera optics models can express more types of spatial distortions, pinhole camera models are more commonly used for planar perspective transformations because they are more complex and not directly related to perspective restoration.

Plane perspective transformation describes the process of projecting a point on a two-dimensional plane in three-dimensional space to another two-dimensional plane. That is to say, the image captured by the camera is a two-dimensional image. This is also assumed by perspective reduction in current QR code applications, where the QR code image is flat.

If the two-dimensional coordinates of a point on the two-dimensional space in the plane two-dimensional code are represented by homogeneous coordinates: x″=[x″y″z″] ^T , that is to say, its two-dimensional Cartesian coordinates are:

For plane perspective transformation, the homogeneous coordinates after transformation are x′=Ax″:

Correspondingly, the Cartesian coordinates of x' in the captured two-dimensional image are:

不会改变。Among them, A is a 3×3 non-singular matrix. In perspective transformation, A is a homogeneous matrix with 8 degrees of freedom, because after multiplying it by a scalar constant,

will not change.

Perspective restoration steps

After getting 6 calibration points, it can be divided into upper and lower parts for processing. Since the degree of freedom of perspective transformation is 8, and the coordinates of each reference point are two-dimensional coordinates, four reference points are needed to estimate a complete homogeneous matrix A.

In the previous step, 6 calibration points were obtained, so for the upper half of the QR code image, the 2 calibration points of the upper half (such as top corner points) and the 2 calibration points of the middle folded part (such as Fold corners) as reference points. For the bottom half, the 2 correction points of the bottom half (eg top corners) and the 2 correction points of the middle folded part (eg fold corners) can be used as reference points. In this way, the folded corners of the middle folded part are used twice in total to estimate two different perspective transformations.

First, assume that the upper and lower parts of the two-dimensional code image are projected from a square of unit length (subsequent processing will be performed to make the lengths of the upper and lower parts of the two-dimensional code match), that is:

(0,0)→(x ₀ ,y ₀ ),

(1,0)→(x ₁ ,y ₁ ),

(0,1)→(x ₂ ,y ₂ ),

(1,1)→(x ₃ ,y ₃ )

Among them, (x _i , y _i ) are the corner coordinates c _i , and the homogeneous matrix A can be obtained by the following relation:

a ₁₁ ＝x ₁ -x ₀ +a ₁₃ x ₁

a ₂₁ ＝x ₃ -x ₀ +a ₂₃ x ₃

a ₃₁ =x ₀

a ₁₂ =y ₁ -y ₀ +a ₁₃ y ₁

a ₂₂ =y ₃ -y ₀ +a ₂₃ y ₃

a ₃₂ =y ₀

definition:

Δx ₁ =x ₁ -x ₂ , Δx ₂ =x ₃ -x ₃ , Δx ₃ =x ₀ -x ₁ +x ₂ -x ₃

Δy ₁ =y ₁ -y ₂ , Δy ₂ =y ₃ -y ₂ , Δy ₃ =y ₀ -y ₁ +y ₂ -y ₃

then it can be calculated

和

来将上下两部分分别还原为正方形图像，如图13A和图13B所示，其示出了得到的透视还原结果。其中，图13A显示了还原后的该折叠二维码图像的第一部分图像(即上部图像)，图13B显示了还原后的该折叠二维码图像的第二部分图像(即下部图像)。Due to the different perspective distortions experienced by the upper and lower parts, the corresponding homogeneous matrices are also different, which can be represented by A ₁ and A ₂ respectively, and the two-dimensional code image is obtained through two sets of different reference points. Then you can pass

and

To restore the upper and lower parts to a square image respectively, as shown in FIG. 13A and FIG. 13B , which show the obtained perspective restoration results. Wherein, FIG. 13A shows the first partial image (ie, the upper image) of the folded two-dimensional code image after restoration, and FIG. 13B shows the second partial image (ie, the lower image) of the folded two-dimensional code image after restoration.

Image Stitching Steps

The image splicing step mainly involves splicing the first part of the image and the second part of the image to obtain a plane two-dimensional code image.

Among them, after obtaining the square perspective restoration results of the upper and lower parts of the folded two-dimensional code image (ie, the first part of the image and the second part of the image), when the width of the restored two-dimensional code image remains unchanged, their lengths need to be adjusted to their original lengths, so that in the last step, the restored two-dimensional code images of the upper and lower parts are fused to obtain the original plane two-dimensional code image.

Module Quantity Estimation

First, calculate the number of modules in the horizontal direction of the captured QR code image through the positioning graphics located at one of the upper and lower ends of the upper and lower parts of the QR code image, that is, the alternating black and white modules, as shown in Figure 14 and shown in Figure 15. It is very easy to confirm which end contains the positioning graphic, because the side that does not contain the positioning graphic will not have alternating black and white modules. At the same time, it is assumed that the modules are square with the same length and width.

As shown in Figure 14, in the black-and-white binary image, the scanning position of the horizontal positioning pattern is determined from the extracted two calibration points at the lower end (that is, the top corner point), which is marked with a line segment "L _h ".

As shown in FIG. 15 , it shows the scanning result of the line segment “L _h ” in FIG. 14 , from which the number of modules in the horizontal direction can be estimated. In Figure 15, each process from 0→1→0 represents a module, and similarly, each process from 1→0→1 also represents a module, so the number n _h of modules in the horizontal direction can be obtained from it. Here, n _h should be an integer, because the two-dimensional code image in the horizontal direction is not divided, and it should contain an integer number of modules.

In the same way, on the right side of the restored upper and lower two-dimensional code images, the number of modules in the vertical direction _nv,upper , _nv , of the upper and lower two-dimensional code images can be determined. _lower . One difference from the horizontal direction is that n _v,upper and n _v,lower may not be integers, because the intersection of the upper and lower parts of the folded two-dimensional code image can be in the middle of a module in the vertical direction, resulting in the upper and lower two Sections each have a portion of this module. Here, the pixel length d _upper,bottom occupied by the lowermost module in the upper part of the two-dimensional code image and the pixel length d _lower,top occupied by the uppermost module in the lower part can be used to obtain:

Among them, _#upper and _{#lower are the numbers of alternating black and white in the vertical positioning graphics of the upper and lower} parts respectively, that is, the integer parts of n _v,upper and n _v,lower .

Image resizing and stitching

In this step, the upper and lower parts of the originally restored QR code image will be adjusted to their original aspect ratio by the square image. Through the width w of the upper and lower parts of the two-dimensional code image after perspective restoration and the number of modules n _h in the horizontal direction estimated in the previous step, the size m of the module after perspective restoration can be calculated:

m=w/n _h

Thus, the length (height) h _upper of the upper part of the two-dimensional code image after perspective restoration can be calculated:

h _upper = n _{v, upper} × m

and the length (height) h _lower of the QR code image in the lower part:

h _lower = n _{v, lower} × m

Therefore, the length (height) of the upper and lower parts, which are originally the same as the width, is adjusted to h _upper and h _lower . The adjusted results are shown in Figure 16A and Figure 16B, where Figure 16A shows the correction obtained after image size adjustment Fig. 16B shows a schematic diagram of the corrected second partial image (ie, the lower image) obtained after image size adjustment.

Finally, splicing the upper and lower parts of the two-dimensional code image to obtain the restored flat two-dimensional code image, as shown in Figure 17. The restored planar two-dimensional code image can be recognized by the two-dimensional code decoder, so that relevant information can be obtained.

As shown in Fig. 18, the present invention accordingly provides a restoration device 180 for folding a two-dimensional code image, which may include:

Image preprocessing unit 181, configured to preprocess a folded two-dimensional code image to obtain a preprocessed image containing two-dimensional code image information;

A corner detection unit 182, configured to perform corner detection on the preprocessed image to obtain multiple correction points of the folded two-dimensional code image, wherein the multiple correction points include at least the top of the folded two-dimensional code image corner points and corner points located on the folding line of the folded two-dimensional code image;

The image restoration unit 183 is configured to perform perspective restoration on the folded two-dimensional code image by using the corner point and the top corner point, so as to obtain the first part image and the second part image of the folded two-dimensional code image in two intersection planes. two-part image; and

The image splicing unit 184 is configured to splice the first partial image and the second partial image to obtain a plane two-dimensional code image.

Preferably, the image preprocessing unit includes a morphological processing module for performing morphological processing on the folded two-dimensional code image, and extracting the two-dimensional code image information in the folded two-dimensional code image by distinguishing different objects , wherein the two-dimensional code image information includes the location information of the two-dimensional code image.

Preferably, the image preprocessing unit may also include:

A grayscale value conversion processing module, configured to perform grayscale value conversion processing on the folded two-dimensional code image to obtain a grayscale value image; and

The binarization processing module is used to perform binarization processing on the folded two-dimensional code image to obtain a black and white binary image.

In the present invention, when the morphological processing module performs morphological processing on the folded two-dimensional code image, it uses structural elements to perform a closing operation on the black and white binary image, so that the boundary of the two-dimensional code image in the black and white binary image becomes and to fill the black and white binary image to distinguish the two-dimensional code image from other objects, and obtain the position information of the two-dimensional code image in the black and white binary image.

Preferably, the corner detection unit may include:

An edge detection module, configured to use the filled black and white binary image as a label image, and perform edge detection on it to obtain multiple edges of the two-dimensional code image, and obtain an edge label image;

A line detection module, configured to perform line detection on the edge label image to extract edge information of the plurality of edges of the two-dimensional code image; and

The corner detection module is configured to use the edge information of the plurality of edges to perform corner detection, so as to obtain a plurality of top corners of the two-dimensional code image and folding corners located on the folding line.

In the present invention, the image restoration unit assumes that the first partial image and the second partial image of the folded two-dimensional code image are both projected from a square of unit length, and the obtained restored first partial image and the second part of the image are both square images.

Preferably, the image splicing unit may include:

The module quantity estimation module is used to calculate the module quantity of the two-dimensional code image in the horizontal direction and the vertical direction respectively by the module quantity estimation method;

The image size adjustment module is used to adjust the restored two-dimensional code image according to the width of the first part of the image and the width of the second part of the image, and the calculated number of modules in the horizontal and vertical directions. The first partial image and the second partial image of the two-dimensional code image are square-scaled to their original aspect ratios to obtain a corrected first partial image and a corrected second partial image; and

The image splicing module is used to splice the corrected first part of the image and the corrected second part of the image to obtain the planar two-dimensional code image.

In the present invention, the module quantity estimation module is:

In the black-and-white binary image, the scanning position of the horizontal positioning figure is determined by the two top corner points obtained in the horizontal direction, and the horizontal position of the two-dimensional code image is determined according to the scanning result of the horizontal positioning figure. number of modules on

In the black-and-white binary image, the scanning position of the vertical positioning figure is determined by the top corner and folding corner obtained in the vertical direction, and the two-dimensional code image is determined according to the scanning result of the vertical positioning figure. The number of modules in the vertical direction.

In the present invention, when the corner point detection step fails to detect or the image restoration step fails to restore, the morphological processing module also increases the size of the structural elements to re-perform the closing operation step.

In the present invention, the image preprocessing unit may further include:

The cutting module is used to obtain a captured image through an image capturing device, and perform cutting processing on the captured image to obtain the folded two-dimensional code image with a predetermined size.

As shown in Figure 19, the present invention also provides a code scanning device 190 for folding a two-dimensional code image, which may include:

A camera unit 191 for capturing a folded two-dimensional code image;

A device 180 for restoring the folded two-dimensional code image as described above, used to restore the folded two-dimensional code image to obtain a plane two-dimensional code image; and

A two-dimensional code decoder 192, configured to decode the restored planar two-dimensional code image.

Aiming at the perspective distortion of folding deformation, the present invention designs a related restoration method and its device and related code scanning equipment without changing the structure of the two-dimensional code of the image itself, so that the planes located on the commodity packaging meet It is possible to extract information from image QR codes pasted everywhere.

Although the present invention has been disclosed above in terms of implementation, it is not intended to limit the present invention. Any skilled person can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the present invention The scope of protection should be based on the scope defined by the appended claims.

Claims (21)

1. A method for restoring a folded two-dimensional code image, comprising: The image preprocessing step is to preprocess a folded two-dimensional code image to obtain a preprocessed image containing two-dimensional code image information; The corner point detection step is to perform corner point detection on the preprocessed image to obtain a plurality of correction points of the folded two-dimensional code image, wherein the plurality of correction points at least include the top corner point of the folded two-dimensional code image and A folding corner located on a folding line of the folding two-dimensional code image; The image restoration step is to use the folded corners and the top corners to perform perspective restoration on the folded two-dimensional code image, so as to obtain the first part image and the second part of the folded two-dimensional code image in two intersection planes image; The image splicing step is to splice the first part of the image and the second part of the image to obtain a plane two-dimensional code image. 2. The method for restoring the folded two-dimensional code image according to claim 1, wherein the image preprocessing step comprises: Performing morphological processing on the folded two-dimensional code image, by distinguishing different objects in the folded two-dimensional code image, extracting the two-dimensional code image information, wherein the two-dimensional code image information includes the position information of the two-dimensional code image . 3. The method for restoring the folded two-dimensional code image according to claim 2, further comprising: performing grayscale conversion processing on the folded two-dimensional code image to obtain a grayscale image; Binarize the gray value image to obtain a black and white binary image. 4. The restoration method of the folded two-dimensional code image according to claim 3, wherein, when performing morphological processing on the folded two-dimensional code image, it includes: performing a closing operation on the black-and-white binary image by using structural elements, so that the boundaries of the two-dimensional code images in the black-and-white binary image become continuous; The black and white binary image is filled to distinguish the two-dimensional code image from other objects, and the position information of the two-dimensional code image in the black and white binary image is obtained. 5. The restoration method of the folded two-dimensional code image according to claim 4, wherein the corner detection step comprises: Use the filled black and white binary image as a label image, and perform edge detection on it to obtain multiple edges of the two-dimensional code image, and obtain an edge label image; performing line detection on the edge label image to extract edge information of the multiple edges of the two-dimensional code image; Corner detection is performed by using the edge information of the plurality of edges to obtain a plurality of top corners and folding corners on the folding line of the two-dimensional code image. 6. The method for restoring a folded two-dimensional code image according to claim 5, characterized in that, in the image restoration step, the obtained restored first part of the image and the second part of the image are both long after correction A square image with an aspect ratio set to 1. 7. The method for restoring the folded two-dimensional code image according to claim 6, wherein the image mosaic step comprises: Calculate the number of modules of the two-dimensional code image in the horizontal direction and the vertical direction respectively by the module number estimation method; According to the width of the first part image and the second part image of the restored two-dimensional code image, and the calculated number of modules in the horizontal direction and the vertical direction, the second part of the restored two-dimensional code image is The part of the image and the second part of the image are adjusted from the square image to their original aspect ratio to obtain the corrected first part of the image and the corrected second part of the image; splicing the corrected first part image and the corrected second part image to obtain the planar two-dimensional code image. 8. The restoration method of the folded two-dimensional code image according to claim 7, wherein the method for estimating the number of modules comprises: In the black-and-white binary image, the scanning position of the horizontal positioning figure is determined by the two top corner points obtained in the horizontal direction, and the horizontal position of the two-dimensional code image is determined according to the scanning result of the horizontal positioning figure. number of modules on In the black-and-white binary image, the scanning position of the vertical positioning figure is determined by the top corner and folding corner obtained in the vertical direction, and the two-dimensional code image is determined according to the scanning result of the vertical positioning figure. The number of modules in the vertical direction. 9. The method for restoring a folded two-dimensional code image according to any one of claims 4 to 8, wherein, when the corner point detection step fails to detect or the image restoration step fails, further comprising: The closing operation step is repeated by increasing the size of the structuring element. 10. The method for restoring the folded two-dimensional code image according to claim 9, further comprising: A captured image is obtained by an image capturing device, and the captured image is cut to obtain the folded two-dimensional code image with a predetermined size. 11. A device for restoring a folded two-dimensional code image, comprising: An image preprocessing unit, configured to preprocess a folded two-dimensional code image to obtain a preprocessed image containing two-dimensional code image information; A corner detection unit, configured to perform corner detection on the preprocessed image to obtain a plurality of correction points of the folded two-dimensional code image, wherein the plurality of correction points include at least the top corner of the folded two-dimensional code image point and the folded corners on the folded line of the folded two-dimensional code image; An image restoring unit, configured to use the folded corners and the top corners to perform perspective restoration on the folded two-dimensional code image, so as to obtain the first part image and the second part of the folded two-dimensional code image in the two intersection planes two-part image; The image splicing unit is used to splice the first partial image and the second partial image to obtain a plane two-dimensional code image. 12. The device for restoring a folded two-dimensional code image according to claim 11, wherein the image preprocessing unit includes: The morphological processing module is used to perform morphological processing on the folded two-dimensional code image, and extract the two-dimensional code image information by distinguishing different objects in the folded two-dimensional code image, wherein the two-dimensional code image information contains two The location information of the QR code image. 13. The device for restoring a folded two-dimensional code image according to claim 12, wherein the image preprocessing unit further comprises: A grayscale value conversion processing module, configured to perform grayscale value conversion processing on the folded two-dimensional code image to obtain a grayscale value image; and The binarization processing module is used to perform binarization processing on the gray value image to obtain a black and white binary image. 14. The device for restoring a folded two-dimensional code image according to claim 13, characterized in that, when the form processing module performs form processing on the folded two-dimensional code image, it uses structural elements to process the black and white binary image Close operation, so that the boundary of the two-dimensional code image in the black and white binary image becomes continuous; and fill the black and white binary image to distinguish the two-dimensional code image from other objects, and obtain the two-dimensional code image in the Position information in black and white binary images. 15. The device for restoring a folded two-dimensional code image according to claim 14, wherein the corner detection unit comprises: An edge detection module, configured to use the filled black and white binary image as a label image, and perform edge detection on it to obtain multiple edges of the two-dimensional code image, and obtain an edge label image; A line detection module, configured to perform line detection on the edge label image, to extract edge information of the plurality of edges of the two-dimensional code image; The corner detection module is configured to use the edge information of the plurality of edges to perform corner detection, so as to obtain a plurality of top corners of the two-dimensional code image and folding corners located on the folding line. 16. The device for restoring a folded two-dimensional code image according to claim 15, characterized in that the restored first part of the image and the second part of the image obtained by the image restoration unit are all corrected with an aspect ratio setting Set to a square image of 1. 17. The device for restoring a folded two-dimensional code image according to claim 16, wherein the image splicing unit comprises: The module quantity estimation module is used to calculate the module quantity of the two-dimensional code image in the horizontal direction and the vertical direction respectively by the module quantity estimation method; The image size adjustment module is used to adjust the restored two-dimensional code image according to the width of the first part of the image and the width of the second part of the image, and the calculated number of modules in the horizontal and vertical directions. The first part image and the second part image of the two-dimensional code image are adjusted to their original aspect ratio by the square image, so as to obtain the corrected first part image and the corrected second part image; The image splicing module is used to splice the corrected first part of the image and the corrected second part of the image to obtain the planar two-dimensional code image. 18. The restoring device of folding two-dimensional code image according to claim 17, is characterized in that, this module number estimation module is: In the black-and-white binary image, the scanning position of the horizontal positioning figure is determined by the two top corner points obtained in the horizontal direction, and the horizontal position of the two-dimensional code image is determined according to the scanning result of the horizontal positioning figure. number of modules on In the black-and-white binary image, the scanning position of the vertical positioning figure is determined by the top corner and folding corner obtained in the vertical direction, and the two-dimensional code image is determined according to the scanning result of the vertical positioning figure. The number of modules in the vertical direction. 19. The device for restoring a folded two-dimensional code image according to any one of claims 14 to 18, characterized in that, when the corner detection step fails or the image restoration step fails, the morphological processing module also adds structure The size of the element, re-perform the closing operation step. 20. The device for restoring a folded two-dimensional code image according to claim 19, wherein the image preprocessing unit further comprises: The cutting module is used to obtain a captured image through an image capturing device, and perform cutting processing on the captured image to obtain the folded two-dimensional code image with a predetermined size. 21. A code scanning device for folding a two-dimensional code image, characterized in that it comprises: A camera unit for capturing a folded two-dimensional code image; A device for restoring a folded two-dimensional code image according to any one of claims 11 to 20, configured to restore the folded two-dimensional code image to obtain a plane two-dimensional code image; A two-dimensional code decoder is used to decode the restored flat two-dimensional code image.

Images