CN115578627B - A monocular image boundary recognition method, device, medium and curtain wall robot - Google Patents

Abstract

The invention provides a monocular image boundary identification method, S10, a first image containing curtain walls is obtained; s20, graying the first image to obtain a gray image; extracting a first edge map from the gray level image by using canny, extracting a second edge map from the first image by using a Laplacian operator, fusing the first edge map and the second edge map, and binarizing the fused first edge map and the second edge map to form a third edge map; s30, acquiring a horizontal line of the third edge map by using a search frame method, and acquiring a far-end roof edge and a near-end horizontal frame; s40, identifying vertical lines in the third edge graph to obtain far-end left and right boundaries and left and right side near-end vertical frames; s50, highlighting the far-end roof edge, the near-end horizontal border, the far-end left-right border, the left-right near-end vertical border and 4 vertical borders by using different color frames. The invention can provide visual auxiliary functions for operators, and avoid the situation that the operators collide with the frame or move out of the boundary due to distraction.

Description

A monocular image boundary recognition method, device, medium and curtain wall robot

technical field

The present invention relates to the technical field of robots, in particular to a monocular image boundary recognition method, device, medium and curtain wall robot.

Background technique

During the design and installation of glass curtain walls in high-rise buildings, different types of window frames and other fixings are installed between the glasses to fix each piece of glass to prevent it from falling off. However, these frame fixtures that ensure the safety of the glass curtain wall have become obstacles to the cleaning, adsorption and travel of the cleaning robot, and have also become a challenge to the operation of the cleaning robot. To solve the problem that the curtain wall frame affects the operation, the curtain wall robot first needs to realize the perception and recognition of the curtain wall frame. Existing methods for curtain wall cleaning robots to identify glass curtain wall boundaries include methods such as ground observation, sensor feedback, and camera visual inspection.

Most of the curtain wall cleaning robots are equipped with collision sensors to prevent the operator from colliding with the curtain wall frame due to improper operation. Collision sensors are generally deployed on the four sides or four corners of the robot. When the robot approaches the frame, the collision sensor first hits the frame and returns an electrical signal. After receiving the signal, the robot stops to prevent collision. However, the collision sensor needs to be hit to return the signal. In order to further improve the observation ability of the operator, some curtain wall cleaning robots are equipped with cameras, and the real-time images are transmitted to the remote control of the ground operator through the image transmission device. The operator observes the picture of the robot's camera and judges the boundary conditions of the robot's forward direction, so as to operate and adjust.

In academia, scholars have studied image recognition algorithms applied to curtain wall cleaning robots to identify and extract boundaries such as window frames on curtain walls, so as to provide perception data for the automatic operation of robots. For example, the Sobel operator is directly used to extract the boundary, the fusion of edge detection and Hough transform algorithm is used to judge whether the edge is reached, and the HSI model is used to identify the curtain wall, etc.

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

Contents of the invention

Aiming at the above-mentioned technical problems in the related art, the present invention proposes a monocular image boundary recognition method, which includes the following steps:

S10. Obtain the first image including the curtain wall;

S20. Grayscale the first image, and obtain a grayscale image after grayscale; use a canny operator to extract a first edge map on the grayscale image, and perform a Laplacian calculation on the first image Sub-extracting the second edge map, merging and binarizing the first edge map and the second edge map to form a third edge map;

S30. Use the search box method to obtain the horizontal line of the third edge map, and obtain the far-end roof edge and the near-end horizontal border; wherein the search box method is to set a rectangular area, if the pixel value in the area is a high value after binarization If the number of pixels is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary and then expand to the left and right in the top and bottom areas, and search the left and right ends to obtain the edge of the roof;

S40, identifying the vertical lines in the third edge map to obtain the far left and right borders and the left and right proximal vertical borders;

S50. Highlight the far-end roof edge, the proximal horizontal border, the far left-right border, the left-right proximal vertical border, and the four vertical borders with boxes of different colors.

Specifically, step S20 also includes:

S21. Using the Canny operator to extract the edge image from the grayscale image to obtain the first edge image;

S22. Using a Gaussian filter on the first image, after performing grayscale conversion, use a Laplacian operator to extract an edge map to obtain a second edge map;

S23. Assign weights of 0.5 to the pixels of the first edge map and the second edge map, and then perform image superimposition;

S24. After the superimposition, assign a value of 0 to those whose grayscale is less than 10, and assign a value of 255 to those whose grayscale is greater than 10.

Specifically, step S30 includes:

S33. Move the search box from top to bottom or from bottom to top according to the set area of the third edge map, and count the number of pixels in the search box whose pixel value is a high value after binarization until the pixel is found for the first time If the number is greater than the set threshold, obtain the top/bottom height value to obtain the far-end roof edge; among them, for searching the far-end roof edge, the width of the search box is 1-5 pixels, and the height is 1 pixel; for searching the near-end horizontal border, The search box has a width of 50% of the image width and a height of 2 pixels;

S34. Continue to move and search until the number of pixels with a pixel value of 255 in the search box is less than the set threshold, and obtain the bottom/top height value to obtain the edge of the far-end roof;

S35 , continue to expand 1-3 pixels according to the top and bottom, start moving from the middle point to both sides, and search for the width values of the left and right ends.

Specifically, step S40 specifically includes using the vertical line search method to identify the vertical lines in the third edge map to obtain the four boundaries of the far-end left and right boundaries and the near-end vertical border, which are specifically:

S41. Remove the pixel values outside the set area from the edge image, and then translate the remaining pixels to the middle of the image according to the position of the area;

S42. Cycle from 1° to 90° or from 90° to 1° according to the set direction, with a step size of 1°, rotate the original image by a certain angle each time, and enhance the image with a binary method;

S43. Using corrosion and dilation, extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, and eliminate non-horizontal lines while also eliminating horizontal lines with a length less than 30 pixels;

S44. Use the search box method to search the boundary from top to bottom or from bottom to top according to the setting; if the boundary is found for the first time, record the initial angle and top/bottom height; if the boundary is found and cannot be found again, record the last angle and bottom/top heights, and break out of the loop;

S45, taking the average value of the first and last angles as the rotation angle;

S46. After rotating according to the angle, search for the left and right ends from the midpoint to both sides in the top and bottom areas;

S47. After acquiring the boundary area, reversely rotate back to the initial state, and translate back to the original position.

In the second aspect, another embodiment of the present invention discloses a monocular image boundary recognition device, which includes the following units:

a curtain wall acquiring unit, configured to acquire the first image containing the curtain wall;

An edge image acquisition unit, configured to grayscale the first image, and obtain a grayscale image after grayscale; use a canny operator to extract a first edge image on the grayscale image, and pull the first image The Placian Laplacian operator extracts the second edge map, and the first edge map and the second edge map are fused and binarized to form the third edge map;

The horizontal line acquisition unit is used to obtain the horizontal line of the third edge map by using the search box method, and obtain the far-end roof edge and the near-end horizontal border; wherein the search box method is to set a rectangular area, if the pixel value in the area is binarized If the number of pixels with the highest value is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary, and then expand to the left and right in the top and bottom areas, and search the left and right ends to obtain the edge of the roof;

a vertical line obtaining unit, configured to identify the vertical lines in the third edge map to obtain the far left and right borders and the left and right proximal vertical borders;

The highlight display unit is configured to highlight the far roof edge, the near horizontal border, the far left and right borders, the left and right proximal vertical borders, and the four vertical borders with boxes of different colors.

Specifically, the edge image acquisition unit also includes:

The canny extraction unit is used to extract the edge map using the Canny operator on the grayscale image to obtain the first edge map;

A Laplacian extraction unit, configured to use Gaussian filtering on the first image, and use a Laplacian operator to extract an edge map after grayscale to obtain a second edge map;

An image superposition unit, which is used to assign a weight of 0.5 to the pixels of the first edge map and the second edge map, and then perform image superposition;

The gray value assignment unit is used to assign the gray value less than 10 to 0 and the gray value greater than 10 to 255 after superposition.

Specifically, the horizontal line acquisition unit also includes:

The search box moving unit is used to move the search box from top to bottom or from bottom to top according to the set area of the third edge map, and count the number of pixels in the search box whose pixel values are binarized high values after the movement , until the number of pixels is found to be greater than the set threshold for the first time, this height is the top/bottom height value to obtain the far-end roof edge; among them, for searching the far-end roof edge, the width of the search box is 1-5 pixels, and the height is 1 pixel , for the search proximal horizontal border, the search box has a width of 50% of the image width and a height of 2 pixels.

The first horizontal line acquisition unit is used to continue to move and search until the number of pixels in the search box whose pixel value is a binarized high value is less than a set threshold, and this height is the bottom/top height value;

The left and right end width value acquisition unit is used to expand 1-3 pixels according to the top and bottom, start to move from the middle point to both sides, and search for the width value of the left and right ends.

Specifically, the vertical line acquisition unit specifically includes using the vertical line search method to identify the vertical line in the third edge map to obtain the four boundaries of the far-end left and right boundaries and the near-end vertical border, which are specifically:

The second pixel removal unit is used to remove the pixel values outside the set area from the edge map, and then translate the remaining pixels to the middle of the image according to the position of the area;

The rotation unit is used to cycle from 1° to 90° or from 90° to 1° according to the set direction, with a step size of 1°, to rotate the original image by a certain angle each time, and to enhance the image with the binary method;

Erosion and expansion unit, used to use erosion and expansion to extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, and eliminate non-horizontal lines and horizontal lines with a length less than 30 pixels;

The boundary acquisition unit is used to use the search box method to search the boundary from top to bottom or from bottom to top according to the setting; if the boundary is found for the first time, record the initial angle and top/bottom height; if the boundary is found and cannot be found again, Then record the last angle and bottom/top height, and jump out of the loop;

The rotation angle acquisition unit is used to take the average value of the first and last angles as the rotation angle;

The left and right end search unit is used to search the left and right ends from the midpoint to both sides in the top and bottom area after rotating according to the angle;

The reverse rotation unit is used to obtain the boundary area, reverse rotation back to the initial state, and translate back to the original position.

In the third aspect, another embodiment of the present invention discloses a curtain wall robot. The curtain wall robot has a monocular camera, a processor, and a memory. Instructions are stored in the memory. When the instructions are executed by the processor, the In order to realize the above-mentioned monocular image boundary recognition method.

In the fourth aspect, another embodiment of the present invention discloses a non-volatile storage medium, where instructions are stored on the non-volatile storage medium, and when the instructions are executed by a processor, they are used to implement the above-mentioned unit A method for image boundary recognition.

The present invention uses the search box method to be insensitive to the influence of noise, and can adapt to the noise well. When the present invention identifies the frame of the curtain wall, according to the different forms of the frame, search boxes of different widths are set to increase the accuracy of recognition of different frames . Further, when identifying the vertical frame of the curtain wall, the present invention rotates the image, and uses the search box method to identify the rotated horizontal line, thereby identifying the nearest/farthest left and right vertical lines, which can remove horizontal lines and oblique lines to a certain extent. line interference. The present invention recognizes 2 horizontal lines and 4 vertical lines, that is, the farthest roof edge, the nearest horizontal frame, the farthest left and right borders, and the nearest vertical frame, and finally these 6 boundary lines are highlighted on the image . Automatically identify the nearest border and the farthest border at the edge of the robot and use a colored box as a reminder. The invention reduces the effort required by operators to observe the boundary, avoids the problem of fuzzy boundary features during image transmission, and only recognizes and prompts key boundaries. The present invention can provide visual auxiliary functions for operators to assist the operators to clean the curtain wall and avoid the situation that the operators bump into the border or move out of the border due to distraction.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a flowchart of a monocular image boundary recognition method provided by an embodiment of the present invention;

Fig. 2 is the original curtain wall image acquired by the monocular camera provided by the embodiment of the present invention;

FIG. 3 is an edge graph obtained by using an edge operator provided by an embodiment of the present invention;

Fig. 4 is a schematic diagram of an extracted horizontal line provided by an embodiment of the present invention;

Fig. 5 is a schematic diagram of the processing process diagram of the lower left area provided by the embodiment of the present invention;

Fig. 6 is a schematic diagram of the highlighted boundary of recognition provided by the embodiment of the present invention;

Fig. 7 is a schematic diagram of a monocular image boundary recognition device provided by an embodiment of the present invention;

Fig. 8 is a schematic diagram of a monocular image boundary recognition device provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention belong to the protection scope of the present invention.

Embodiment one

With reference to Fig. 1, the present embodiment discloses a kind of monocular image boundary recognition method, and it comprises the following steps:

S10. Obtain the first image including the curtain wall;

Referring to FIG. 2 , the monocular image boundary recognition method of this embodiment is applied to a curtain wall robot, and the curtain wall robot is equipped with a monocular camera. When the curtain wall robot cleans the curtain wall, the monocular camera mounted on the curtain wall robot takes photos or videos to form the first image. The first image includes a curtain wall.

S20. Grayscale the first image, and obtain a grayscale image after grayscale; use a canny operator to extract a first edge map on the grayscale image, and perform a Laplacian calculation on the first image Sub-extracting the second edge map, merging and binarizing the first edge map and the second edge map to form a third edge map;

In this embodiment, according to the characteristics of the curtain wall image from the perspective of the robot, the Canny operator and the Laplacian operator are fused to implement a more suitable edge acquisition algorithm. In the edge map fused by the Canny operator and the Laplacian operator, the clear boundary will be thicker, and the unclear boundary will be thinner. Among them, the ones that are clear are those that are relatively close or that change greatly like the boundary. Combining the method of extracting line segments by rotation in this embodiment, the key edge line can be extracted quickly with a larger step size, or the edge line can be extracted more slowly but comprehensively with a smaller step size, and the extraction strategy is more flexible.

Referring to Figure 3, the principle of this step is to use the different characteristics of the two operators to improve the edge recognition accuracy and reduce noise. The Canny operator has clear edges and low noise, but the recognition is incomplete; the Laplacian operator has more edge information, but also has more noise. Therefore, in this embodiment, the parameters of the two operators are set, each of which is assigned a weight of 0.5 and then superimposed, and the two operators are complementary to obtain the edge as completely as possible.

Concrete step S20 also includes:

S21. Using the Canny operator to extract the edge image from the grayscale image to obtain the first edge image;

S22. Using a Gaussian filter on the first image, after performing grayscale conversion, use a Laplacian operator to extract an edge map to obtain a second edge map;

S23. Assign weights of 0.5 to the pixels of the first edge map and the second edge map, and then perform image superimposition;

S24. After the superimposition, assign a value of 0 to those whose grayscale is less than 10, and assign a value of 255 to those whose grayscale is greater than 10.

After the implementation of superimposition, the value of grayscale less than 10 is assigned as 0, and the value of grayscale greater than 10 is assigned as 255, and the edge map is enhanced by binarization method.

S30. Use the search box method to obtain the horizontal line of the third edge map, and obtain the far-end roof edge and the near-end horizontal border; wherein the search box method is to set a rectangular area, if the pixel value in the area is a high value after binarization If the number of pixels is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary and then expand to the left and right in the top and bottom areas, and search the left and right ends to obtain the edge of the roof;

After performing grayscale binarization on the third edge image in step S20, 255 represents edge points, and 0 represents non-edge points. In this embodiment, the highest value 255 after binarization represents the edge point, and the lowest value 0 represents the non-edge point as an example. Those skilled in the art can use other pixel values to distinguish the edge point from the non-edge point. In this implementation Examples are not repeated here.

For the horizontal line, it is generally located on an edge point. Therefore, in this embodiment, when enough edge points are searched, it means that there is a straight line in the search area. Therefore, this embodiment uses the number of pixels with a pixel value of 255 in the area greater than the threshold to determine whether the boundary is found. In this embodiment, the border of the frame of the curtain wall is generally a horizontal line.

This step is to use the search box method to identify the two boundaries of the far-end roof edge and the near-end horizontal frame.

The principle of the search box method is to set a rectangular area. If the number of pixels with a high value after binarization in the area is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary. It then expands left and right within the top and bottom areas, searching for the left and right ends. Compared with single pixel search, the method of this embodiment has high search efficiency and fast speed, and can remove noise.

Concrete step S30 comprises:

S33. Move the search box from top to bottom or from bottom to top according to the set area of the third edge map, count the number of pixels with a pixel value of 255 in the search box after the movement, until the number of pixels is found to be greater than the set threshold for the first time , the height is the top/bottom height value to get the far roof edge; where, for searching the far roof edge, the search box has a width of 1-5 pixels and a height of 1 pixel, and for searching the near horizontal border, the search box has The width is 50% of the image width and the height is 2 pixels;

Move the search box from top to bottom or from bottom to top according to the set area of the third edge map, count the number of pixels with a pixel value of 255 in the search box after the movement, until the number of pixels is found to be greater than the set threshold for the first time, the The height is the top/bottom height value to obtain the far roof edge; among them, for searching the far roof edge, since the far edge in the third edge map will be very thin and narrow, the width of the search box is 1-5 pixels , with a height of 1 pixel, use a thinner and shorter search box to increase the probability of finding the edge; for searching the near-end horizontal border, since the closer and more obvious edges in the third edge image will be longer and thicker, use a width of 50% The width and height of the image are 2 pixels to remove obstacles that are not horizontal borders;

S34. Continue to move and search until the number of pixels with a pixel value of 255 in the search box is less than the set threshold, and this height is the bottom/top height value;

S35, continue to expand 1-3 pixels according to the top and bottom, start moving from the middle point to both sides, and search for the width value of the left and right ends;

In this embodiment, extending 1-3 pixels from the top and the bottom can reduce the edge line protruding/recessing by 1 pixel, or the impact caused by the edge line shifting up and down caused by a relatively small roll angle.

Specifically, the horizontal line in this embodiment is equivalent to a rectangle whose height and width are not fixed, so the Y values of the top and bottom ends of the horizontal line are obtained through S33 and S34, and the left and right X values are obtained through S35, that is, the four sides of the rectangle .

Take the example of searching for a proximal horizontal border.

S33 The search box moves upward from the bottom of the image, and it is found that the number of pixels is greater than the threshold for the first time. At this time, the height of the search box is the bottom Y bottom of the horizontal line;

S34 The search box continues to move upward, and the pixel value of the search box will always be greater than the threshold when it is inside the horizontal line. When the pixel value of the search box is found to be less than the threshold, it means that the search box has reached the top Y _top of the horizontal line;

S35 The search box searches from the middle to the left, and when it is inside the horizontal line, the pixels of the search box will always be greater than the threshold. When it is found that the pixels of the search box are smaller than the threshold, it means that the search box has reached _the left end X of the horizontal line:

When the search reaches the left end of the horizontal line X _left , the search box starts to move from the middle to the right again. When the pixel of the search box is found to be smaller than the threshold, it means that the search box reaches the right end X _right of the horizontal line;

Therefore, the coordinates of the four apex corners of the rectangle of the horizontal line are (X _left , Y _top ), (X _right , Y _top ), (X _left , Y _bottom ), (X _right , Y _bottom ).

Referring to Figure 4, the edge of the far roof is focused on the upper and lower 5% of the midpoint of the image. From top to bottom, press the search box with a width of 3 pixels, a height of 1 pixel, and a threshold of 1 pixel to obtain the top and bottom of the boundary, and obtain After the left and right ends, return the top and bottom height values and left and right width values of the roof edge.

The near-end horizontal frame focuses on the area from the bottom of the roof edge to the bottom of the image, and crops 25% of the left and right. Search from bottom to top according to the search box with the roof edge width, 2 pixel height, and 40% search box pixel threshold, and return A rectangular box with a 2px extension at the top and bottom.

In order to remove the influence of noise in the third edge image, step S30 also includes:

S31. Remove pixel values outside the set area of the third edge map;

S32. Using erosion and dilation, extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, so as to obtain an edge map after noise removal;

Specifically, in step S31, the third edge map is replaced with the edge map after noise removal.

While eliminating non-horizontal lines, horizontal lines with a length less than 30 pixels are also eliminated;

S40, identifying the vertical lines in the third edge map to obtain the far left and right borders and the left and right proximal vertical borders;

This step is to use the vertical line search method to identify the four borders of the far left and right borders and the left and right proximal vertical borders.

Referring to Figure 5, the vertical line search method is based on the principle that the vertical lines projected on the left and right ends of the robot will present a certain angle on the image. After focusing on the target area, gradually rotate the image in the specified direction, and use the search box method to identify the rotated horizontal line. The nearest/farthest left and right vertical lines are identified. This method can remove the interference of horizontal lines and oblique lines to a certain extent. Specific steps are as follows:

S41. Remove the pixel values outside the set area from the edge image, and then translate the remaining pixels to the middle of the image according to the position of the area;

S42. Cycle from 1° to 90° or from 90° to 1° according to the set direction, with a step size of 1°, rotate the original image by a certain angle each time, and enhance the image with a binary method;

In this embodiment, the cycle is performed with a step size of 1°, taking into account both precision and operation time. When the step size is reduced, for example, to 0.5°, the accuracy of the first and last angles can be increased, thereby improving the accuracy of the rotation angle, but the calculation time will be increased; when the step size is increased, for example, to 3°, the calculation time can be reduced, And eliminate the probability of searching for a farther and less obvious frame, but at the same time it is also prone to not being able to search for a near-end frame, and the accuracy of the rotation angle is low.

S43. Using corrosion and dilation, extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, and eliminate non-horizontal lines while also eliminating horizontal lines with a length less than 30 pixels;

S44. Use the search box method to search the boundary from top to bottom or from bottom to top according to the setting. If the boundary is found for the first time, record the initial angle and top/bottom height. If the boundary is found and it is not found again, record the last angle and bottom/top height, and jump out of the loop;

S45, taking the average value of the first and last angles as the rotation angle;

S46. After rotating according to the angle, search for the left and right ends from the midpoint to both sides in the top and bottom areas;

S47. After acquiring the boundary area, reversely rotate back to the initial state, and translate back to the original position.

The right far border and right near border are the regions that extract the lower right corner of the image, from the left side of the roof edge to the width of the right border of the image, and from the top of the roof edge to the bottom of the image. The image of the far border on the right is rotated counterclockwise from 1° to 90°, and is identified by the search box method from top to bottom; the image of the proximal border on the right is rotated counterclockwise from 90° to 1°, and identified by the The search box method is used for identification.

The left far border and left near border are to extract the lower left corner of the image, from the left border of the image to the wide right side of the roof edge, and from the top of the roof edge to the bottom of the image. The image of the far border on the left is rotated clockwise from 1° to 90°, and is identified by the search box method from top to bottom; the image of the proximal border on the left is rotated clockwise from 90° to 1°, and identified by the The search box method is used for identification.

S50. Highlight the far-end roof edge, the proximal horizontal border, the far left-right border, the left-right proximal vertical border, and the four vertical borders with boxes of different colors.

Referring to Figure 6, this step is to highlight 2 horizontal and 4 vertical borders with different color boxes to remind the operator.

In this embodiment, the search box method is not sensitive to the influence of noise, and can adapt to noise well. In this embodiment, when identifying the frame of the curtain wall, according to the different shapes of the frame, search boxes of different widths are set to increase the recognition efficiency of different frames. accuracy. Further, when identifying the vertical frame of the curtain wall, this embodiment rotates the image, and uses the search box method to identify the rotated horizontal line, so as to identify the nearest/farthest left and right vertical lines, which can remove horizontal lines and Interference with slashes. This embodiment recognizes 2 horizontal lines and 4 vertical lines, that is, the farthest edge of the roof, the closest horizontal border, the farthest left and right borders, and the closest vertical border, and finally these 6 borders are highlighted on the image Wire. Automatically identify the nearest border and the farthest border at the edge of the robot and use a colored box as a reminder. The invention reduces the effort required by operators to observe the boundary, avoids the problem of fuzzy boundary features during image transmission, and only recognizes and prompts key boundaries. The present invention can provide visual auxiliary functions for operators to assist the operators to clean the curtain wall and avoid the situation that the operators bump into the border or move out of the border due to distraction.

Further, the monocular image boundary recognition method of this embodiment uses different search box sizes for the far-end and near-end horizontal borders, which can better adapt to the thin and narrow edges at the far end and the narrow borders at the near end. The edges are long and thick, which improves the accuracy of edge recognition. In addition, the frame is a relatively thin edge in the image. For example, compared to the lane line on the road, the frame used for the curtain wall in this embodiment is very thin. This embodiment uses different search boxes The size of can be better adapted to the border recognition of the curtain wall. Furthermore, the method for identifying the frame of the curtain wall needs to run on the robot, which has requirements for power consumption and performance. This application can reduce power consumption and achieve relative better performance.

Embodiment two

Referring to Fig. 7, the present embodiment discloses a monocular image boundary recognition device, which includes the following units:

a curtain wall acquiring unit, configured to acquire the first image containing the curtain wall;

Referring to FIG. 2 , the monocular image boundary recognition method of this embodiment is applied to a curtain wall robot, and the curtain wall robot is equipped with a monocular camera. When the curtain wall robot cleans the curtain wall, the monocular camera mounted on the curtain wall robot takes photos or videos to form the first image. The first image includes a curtain wall.

An edge image acquisition unit, configured to grayscale the first image, and obtain a grayscale image after grayscale; use a canny operator to extract a first edge image on the grayscale image, and pull the first image The Placian Laplacian operator extracts the second edge map, and the first edge map and the second edge map are fused and binarized to form the third edge map;

In this embodiment, according to the characteristics of the curtain wall image from the perspective of the robot, the Canny operator and the Laplacian operator are fused to implement a more suitable edge acquisition algorithm.

Referring to FIG. 3 , the principle of the edge image acquisition unit in this embodiment is to use different characteristics of the two operators to improve edge recognition accuracy and reduce noise. The Canny operator has clear edges and low noise, but the recognition is incomplete; the Laplacian operator has more edge information, but also has more noise. Therefore, in this embodiment, the parameters of the two operators are set, each of which is assigned a weight of 0.5 and then superimposed, and the two operators are complementary to obtain the edge as completely as possible.

The specific edge image acquisition unit also includes:

The canny extraction unit is used to extract the edge map using the Canny operator on the grayscale image to obtain the first edge map;

A Laplacian extraction unit, configured to use Gaussian filtering on the first image, and use a Laplacian operator to extract an edge map after grayscale to obtain a second edge map;

An image superposition unit, which is used to assign a weight of 0.5 to the pixels of the first edge map and the second edge map, and then perform image superposition;

The gray value assignment unit is used to assign the gray value less than 10 to 0 and the gray value greater than 10 to 255 after superposition.

After the implementation of superimposition, the value of grayscale less than 10 is assigned as 0, and the value of grayscale greater than 10 is assigned as 255, and the edge map is enhanced by binarization method.

The horizontal line acquisition unit is used to obtain the horizontal line of the third edge map by using the search box method, and obtain the far-end roof edge and the near-end horizontal border; wherein the search box method is to set a rectangular area, if the pixel value in the area is binarized If the number of pixels with the highest value is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary, and then expand to the left and right in the top and bottom areas, and search the left and right ends to obtain the edge of the roof;

In this embodiment, the search box method is used to identify the two boundaries of the far-end roof edge and the near-end horizontal frame.

The principle of the search box method is to set a rectangular area. If the number of pixels with a high value after binarization in the area is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary. It then expands left and right within the top and bottom areas, searching for the left and right ends. Compared with single pixel search, the method of this embodiment has high search efficiency and fast speed, and can remove noise.

The specific horizontal line acquisition unit also includes:

The search box moving unit is used to move the search box from top to bottom or from bottom to top according to the set area of the third edge map, and count the number of pixels with a pixel value of 255 in the search box after the movement until the number of pixels is found for the first time Greater than the set threshold, the height is the top/bottom height value to obtain the far-end roof edge; among them, for searching the far-end roof edge, the width of the search box is 1-5 pixels, and the height is 1 pixel; for searching the near-end level Border, the width of the search box is 50% of the image width, and the height is 2 pixels;

The first horizontal line acquisition unit is used to continue to move and search until the number of pixels with a pixel value of 255 in the search box is less than the set threshold, and this height is the bottom/top height value;

Left and right end width value acquisition unit, used to continue to expand 1-3 pixels according to the top and bottom, start moving from the middle point to both sides, and search for the width value of the left and right ends;

Referring to Figure 4, the edge of the far roof is focused on the upper and lower 5% of the midpoint of the image. From top to bottom, press the search box with a width of 3 pixels, a height of 1 pixel, and a threshold of 1 pixel to obtain the top and bottom of the boundary, and obtain After the left and right ends, return the top and bottom height values and left and right width values of the roof edge.

The near-end horizontal frame focuses on the area from the bottom of the roof edge to the bottom of the image, and crops 25% of the left and right. Search from bottom to top according to the search box with the roof edge width, 2 pixel height, and 40% search box pixel threshold, and return A rectangular box with a top and bottom extension of 2 pixels.

In order to remove the influence of noise in the third edge image, the horizontal line acquisition unit also includes:

a pixel value removal unit, configured to remove pixel values outside the set area of the third edge map;

The noise removal unit is used to use erosion and dilation to extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, so as to obtain the edge map after noise removal;

Specifically, in the search box moving unit, the third edge map is replaced with the edge map after noise removal.

While eliminating non-horizontal lines, horizontal lines with a length less than 30 pixels are also eliminated;

a vertical line obtaining unit, configured to identify the vertical lines in the third edge map to obtain the far left and right borders and the left and right proximal vertical borders;

This step is to use the vertical line search method to identify the four borders of the far left and right borders and the left and right proximal vertical borders.

Referring to Figure 5, the vertical line search method is based on the principle that the vertical lines projected on the left and right ends of the robot will present a certain angle on the image. After focusing on the target area, the image is gradually rotated in the specified direction, and the search box method is used to identify the rotated horizontal line. The nearest/farthest left and right vertical lines are identified. This method can remove the interference of horizontal lines and oblique lines to a certain extent. Specific steps are as follows:

The second pixel removal unit is used to remove the pixel values outside the set area from the edge map, and then translate the remaining pixels to the middle of the image according to the position of the area;

The rotation unit is used to cycle from 1° to 90° or from 90° to 1° according to the set direction, with a step size of 1°, to rotate the original image by a certain angle each time, and to enhance the image with the binary method;

Erosion and expansion unit, used to use erosion and expansion to extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, and eliminate non-horizontal lines and horizontal lines with a length less than 30 pixels;

The boundary acquisition unit is used to use the search box method to search the boundary from top to bottom or from bottom to top according to the setting; if the boundary is found for the first time, record the initial angle and top/bottom height; if the boundary is found and cannot be found again, Then record the last angle and bottom/top height, and jump out of the loop;

The rotation angle acquisition unit is used to take the average value of the first and last angles as the rotation angle;

The left and right end search unit is used to search the left and right ends from the midpoint to both sides in the top and bottom area after rotating according to the angle;

The reverse rotation unit is used to obtain the boundary area, reverse rotation back to the initial state, and translate back to the original position.

The right far border and right near border are the regions that extract the lower right corner of the image, from the left side of the roof edge to the width of the right border of the image, and from the top of the roof edge to the bottom of the image. The image of the far border on the right is rotated counterclockwise from 1° to 90°, and is identified by the search box method from top to bottom; the image of the proximal border on the right is rotated counterclockwise from 90° to 1°, and identified by the The search box method is used for identification.

The left far border and left near border are to extract the lower left corner of the image, from the left border of the image to the wide right side of the roof edge, and from the top of the roof edge to the bottom of the image. The image of the far border on the left is rotated clockwise from 1° to 90°, and is identified by the search box method from top to bottom; the image of the proximal border on the left is rotated clockwise from 90° to 1°, and identified by the The search box method is used for identification.

S50. Highlight the far-end roof edge, the proximal horizontal border, the far left-right border, the left-right proximal vertical border, and the four vertical borders with boxes of different colors.

Referring to Figure 6, this step is to highlight 2 horizontal and 4 vertical borders with different color boxes to remind the operator.

In this embodiment, the search box method is not sensitive to the influence of noise, and can adapt to noise well. In this embodiment, when identifying the frame of the curtain wall, according to the different shapes of the frame, search boxes of different widths are set to increase the recognition efficiency of different frames. accuracy. Further, when identifying the vertical frame of the curtain wall, this embodiment rotates the image, and uses the search box method to identify the rotated horizontal line, so as to identify the nearest/farthest left and right vertical lines, which can remove horizontal lines and Interference with slashes. This embodiment recognizes 2 horizontal lines and 4 vertical lines, that is, the farthest edge of the roof, the closest horizontal border, the farthest left and right borders, and the closest vertical border, and finally these 6 borders are highlighted on the image Wire. Automatically identify the nearest border and the farthest border at the edge of the robot and use a colored box as a reminder. The invention reduces the effort required by operators to observe the boundary, avoids the problem of fuzzy boundary features during image transmission, and only recognizes and prompts key boundaries. The present invention can provide visual auxiliary functions for operators to assist the operators to clean the curtain wall and avoid the situation that the operators bump into the border or move out of the border due to distraction.

Embodiment Three

Referring to FIG. 8 , FIG. 8 is a schematic structural diagram of a monocular image boundary recognition device in this embodiment. The monocular image boundary recognition device 20 of this embodiment includes a processor 21 , a memory 22 and a computer program stored in the memory 22 and operable on the processor 21 . The steps in the foregoing method embodiments are implemented when the processor 21 executes the computer program. Alternatively, when the processor 21 executes the computer program, it realizes the functions of the modules/units in the above device embodiments.

Exemplarily, the computer program can be divided into one or more modules/units, and the one or more modules/units are stored in the memory 22 and executed by the processor 21 to complete this invention. The one or more modules/units may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the computer program in the monocular image boundary recognition device 20 . For example, the computer program can be divided into various modules in Embodiment 2. For the specific functions of each module, please refer to the working process of the device described in the above embodiment, which will not be repeated here.

The monocular image boundary recognition device 20 may include, but not limited to, a processor 21 and a memory 22 . Those skilled in the art can understand that the schematic diagram is only an example of the monocular image boundary recognition device 20, and does not constitute a limitation to the monocular image boundary recognition device 20, and may include more or less components than those shown in the illustration, or Combining certain components, or different components, for example, the monocular image boundary recognition device 20 may also include an input and output device, a network access device, a bus, and the like.

The processor 21 can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processor can be microprocessor or this processor also can be any conventional processor etc., described processor 21 is the control center of described monocular image boundary recognition device 20, utilizes various interfaces and lines to connect whole monocular Various parts of the target image boundary recognition device 20.

The memory 22 can be used to store the computer programs and/or modules, and the processor 21 runs or executes the computer programs and/or modules stored in the memory 22, and calls the data stored in the memory 22, Various functions of the monocular image boundary recognition device 20 are realized. The memory 22 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playback function, an image playback function, etc.) and the like; the storage data area may store Stores data (such as audio data, phonebook, etc.) created according to the use of the mobile phone, etc. In addition, the memory 22 can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card (Flash Card), at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

Wherein, if the integrated modules/units of the monocular image boundary recognition device 20 are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention realizes all or part of the processes in the methods of the above embodiments, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer When the program is executed by the processor 21, the steps of the above-mentioned various method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-OnlyMemory), Random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, computer-readable media Excludes electrical carrier signals and telecommunication signals.

It should be noted that the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physically separated. A unit can be located in one place, or it can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines. It can be understood and implemented by those skilled in the art without creative effort.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (10)

1. A monocular image boundary recognition method, comprising the steps of: S10. Obtain the first image including the curtain wall; S20. Grayscale the first image, and obtain a grayscale image after grayscale; use a canny operator to extract a first edge map on the grayscale image, and perform a Laplacian calculation on the first image Sub-extracting the second edge map, merging and binarizing the first edge map and the second edge map to form a third edge map; S30. Use the search box method to obtain the horizontal line of the third edge map, and obtain the far-end roof edge and the near-end horizontal border; wherein the search box method is to set a rectangular area, if the pixel value in the area is a high value after binarization If the number of pixels is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary, and then expand the search to the left and right in the top and bottom areas to obtain the left and right ends of the top and bottom, and then obtain the edge of the roof, where The border is a horizontal line; S40. Identify vertical lines in the third edge map to obtain far left and right borders and left and right proximal vertical borders. 2. The method of claim 1, further comprising: S50. Highlight the far-end roof edge, the proximal horizontal border, the far left-right border, the left-right proximal vertical border, and the four vertical borders with boxes of different colors. 3. The method according to claim 1, step S20 further comprising: S21. Using the Canny operator to extract the edge image from the grayscale image to obtain the first edge image; S22. Using a Gaussian filter on the first image, after performing grayscale conversion, use a Laplacian operator to extract an edge map to obtain a second edge map; S23. Weighted fusion of pixels of the first edge image and the second edge image, and then perform image superimposition to obtain a third edge image. 4. The method according to claim 1, step S30 comprising: S33. Move the search box from top to bottom or from bottom to top according to the set area of the third edge map, and count the number of pixels in the search box whose pixel value is a high value after binarization until the pixel is found for the first time If the number is greater than the set threshold, obtain the top/bottom height value to obtain the far-end roof edge; among them, for searching the far-end roof edge, the width of the search box is 3-5 pixels, and the height is 1 pixel; for the search for the near-end horizontal border, The search box has a width of 50% of the image width and a height of 2 pixels; S34. Continue to move and search until the number of pixels in the search frame whose pixel value is a binarized high value is less than the set threshold, and obtain the height value of the bottom/top to obtain the edge of the far-end roof; S35 , continue to expand 1-3 pixels according to the top and bottom, start moving from the middle point to both sides, and search for the width values of the left and right ends. 5. The method according to claim 1, step S40 specifically includes using a vertical line search method to identify the vertical line in the third edge figure to obtain the far-end left and right boundaries and the near-end vertical border 4 boundaries, which are specifically: S41. Remove the pixel values outside the set area from the edge image, and then translate the remaining pixels to the middle of the image according to the position of the area; S42. According to the set direction from 1° to 90° or from 90° to 1°, cycle with the preset degree as the step size, rotate the original image by a certain angle each time, and enhance the image with a binary method; S43. Using corrosion and dilation, extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, and eliminate non-horizontal lines while also eliminating horizontal lines with a length less than 30 pixels; S44. Use the search box method to search the boundary from top to bottom or from bottom to top according to the setting; if the boundary is found for the first time, record the initial angle and top/bottom height; if the boundary is found and cannot be found again, record the last angle and bottom/top heights, and break out of the loop; S45, taking the average value of the first and last angles as the rotation angle; S46. After rotating according to the rotation angle, search for the left and right ends from the midpoint to both sides in the top and bottom areas; S47. After acquiring the boundary area, reversely rotate back to the initial state, and translate back to the original position. 6. A monocular image boundary recognition device, which comprises the following units: a curtain wall acquiring unit, configured to acquire the first image containing the curtain wall; An edge image acquisition unit, configured to grayscale the first image, and obtain a grayscale image after grayscale; use a canny operator to extract a first edge image on the grayscale image, and pull the first image The Placian Laplacian operator extracts the second edge map, and the first edge map and the second edge map are fused and binarized to form the third edge map; The horizontal line acquisition unit is used to obtain the horizontal line of the third edge map by using the search box method, and obtain the far-end roof edge and the near-end horizontal border; wherein the search box method is to set a rectangular area, if the pixel value in the area is binarized If the number of pixels with the highest value is greater than the threshold, it is considered that the boundary has been searched, so as to obtain the top and bottom of the boundary, and then expand to the left and right in the top and bottom areas, and search the left and right ends to obtain the edge of the roof; A vertical line acquiring unit, configured to identify the vertical lines in the third edge map to acquire the far left and right borders and the left and right proximal vertical borders. 7. The device according to claim 6, the horizontal line acquisition unit, further comprising: The search box moving unit is used to move the search box from top to bottom or from bottom to top according to the set area of the third edge map, and count the number of pixels in the search box whose pixel values are binarized high values after the movement , until the number of pixels is found to be greater than the set threshold for the first time, the top/bottom height value is obtained to obtain the far-end roof edge; among them, for searching the far-end roof edge, the width of the search box is 3-5 pixels, and the height is 1 pixel; End horizontal border, then the width is 50% of the image width and the height is 2 pixels; The first horizontal line acquisition unit is used to continue to move and search until the number of pixels in the search box whose pixel value is a binarized high value is less than the set threshold, and acquire the bottom/top height value; The left and right end width value acquisition unit is used to expand 1-3 pixels according to the top and bottom, start to move from the middle point to both sides, and search for the width value of the left and right ends. 8. The device according to claim 6, the vertical line acquisition unit specifically includes using a vertical line search method to identify the vertical line in the third edge map to obtain the far-end left and right boundaries and the near-end vertical frame 4 boundaries, which specifically for: The second pixel removal unit is used to remove the pixel values outside the set area from the edge map, and then translate the remaining pixels to the middle of the image according to the position of the area; The rotation unit is used to cycle from 1° to 90° or from 90° to 1° according to the set direction, with the preset degree as the step size, rotate the original image by a certain angle each time, and use the binary method to enhance the image; Erosion and expansion unit, used to use erosion and expansion to extract horizontal lines according to the shape of 30 pixels long and 1 pixel high, and eliminate non-horizontal lines and horizontal lines with a length less than 30 pixels; The boundary acquisition unit is used to use the search box method to search the boundary from top to bottom or from bottom to top according to the setting; if the boundary is found for the first time, record the initial angle and top/bottom height; if the boundary is found and cannot be found again, Then record the last angle and bottom/top height, and jump out of the loop; The rotation angle acquisition unit is used to take the average value of the first and last angles as the rotation angle; The left and right end search unit is used to search the left and right ends from the midpoint to both sides in the top and bottom area after rotating according to the angle; The reverse rotation unit is used to reversely rotate back to the initial state and translate back to the original position after obtaining the boundary area. 9. A curtain wall robot, the curtain wall robot has a monocular camera, a processor, and a memory, and instructions are stored on the memory, and when the instructions are executed by the processor, in order to realize any one of claims 1-5 The monocular image boundary recognition method described in the item. 10. A non-volatile storage medium, wherein instructions are stored on the non-volatile storage medium, and when the instructions are executed by a processor, they are used to implement the unit according to any one of claims 1-5. A method for image boundary recognition.

Images