CN114708422B - A method and device for calculating cabin door coordinates based on binocular images - Google Patents

Abstract

The present invention discloses a method and device for calculating the coordinates of a cabin door based on a binocular image, wherein the method comprises: collecting a binocular image; inputting the binocular image into a first neural network model for first feature extraction, and obtaining a cabin door area sub-map by boundary regression; inputting the cabin door area sub-map into a second neural network model for second feature extraction, and performing a logistic regression on each pixel point of the cabin door area sub-map to obtain a cabin door seam edge map; extracting the coordinates of the edge points based on the cabin door seam edge map, and calculating two two-dimensional coordinate points of the cabin door; and obtaining two three-dimensional coordinate points of the cabin door by triangulation calculation based on the two two-dimensional coordinate points. The present invention can accurately calculate the coordinates of the aircraft cabin door relative to the corridor bridge, and guide the corridor bridge to automatically dock with the aircraft cabin door, greatly improving the degree of automation of the airport, and has strong theoretical significance and practical value.

Description

A method and device for calculating cabin door coordinates based on binocular images

Technical Field

The present invention relates to the technical field of coordinate calculation, and in particular to a method and device for calculating hatch coordinates based on binocular images.

Background technique

With the development of modern society, air travel has become an indispensable part of people's daily life. In the process of serving passengers at the airport, boarding bridge guidance is an indispensable step. The boarding bridge refers to the movable lifting passage connecting the terminal and the cabin in the airport. Before passengers get on and off the plane, the airport needs to guide one end of the boarding bridge to dock with the aircraft's cabin door so that passengers can enter the cabin through the bridge. Figure 1 shows the docking scene of the boarding bridge.

At present, almost all domestic airports use the method of manually driving the jet bridge. The driver uses the naked eye to judge the relative position relationship between the jet bridge and the aircraft, manipulates the jet bridge and docks it to the aircraft door. The manual method cannot accurately obtain the relative coordinates between the jet bridge and the aircraft door, so the docking rate is low. At the same time, training and recruiting experienced drivers will bring high labor costs to the airport. In addition, due to the limitations of the driver's judgment ability, the manual docking method is also prone to accidents.

At present, most domestic airports use manual driving to complete docking, and there is no commercially available method to calculate the coordinates of the aircraft door. The attempts of the industry and academia to develop an automatic docking system for the bridge guidance are divided into two methods: laser ranging method and image comparison method. The laser ranging method refers to installing laser ranging devices around the bridge to measure the distance between the aircraft and the bridge, thereby assisting the pilot to dock with the bridge; the image comparison method refers to matching the aircraft door image with the image in the database to find the aircraft door area in the image and guide the bridge to dock with the aircraft.

The manual operation of the bridge cannot accurately obtain the relative coordinates between the bridge and the aircraft door, so the docking is low. At the same time, training and recruiting experienced pilots will bring high labor costs to the airport. In addition, due to the limitations of the driver's judgment ability, the manual docking method is also prone to accidents.

The disadvantages of the laser ranging method include two aspects. On the one hand, the relative position coordinates between the cabin door and the corridor bridge are composed of three values in the horizontal direction, vertical direction and forward direction, which correspond to the up and down rotation, left and right rotation, and forward and backward rotation of the corridor bridge respectively. The laser ranging method can only obtain the distance in the forward direction, so it can only control the forward and backward movement of the corridor bridge, and the driver still needs to observe to control the up and down left and right rotation of the corridor bridge. On the other hand, the laser sensor is fixed on the corridor bridge, and the angle of laser emission changes with the movement of the corridor bridge, and the area irradiated on the aircraft is also changing. The distance value obtained is not the distance between a fixed point on the aircraft and the corridor bridge, so it can only be used as a reference value to guide the docking of the corridor bridge.

The disadvantages of the image comparison method include two aspects. On the one hand, the image comparison method must build a database of aircraft cabin door images. There are many types of aircraft and new types of aircraft are added every year. Maintaining such a database requires a lot of effort. On the other hand, the image comparison method can only obtain the approximate relative position relationship between the aircraft cabin door and the jet bridge, but cannot obtain the accurate coordinate value, and its docking accuracy is relatively low.

Summary of the invention

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

To this end, the purpose of the present invention is to propose a method for calculating the coordinates of an aircraft door based on binocular images, which can accurately calculate the coordinates of the aircraft door relative to the jet bridge and guide the jet bridge to automatically dock with the aircraft door, greatly improving the degree of automation at the airport, and has strong theoretical significance and practical value.

Another object of the present invention is to provide a device for calculating the coordinates of an aircraft door based on binocular images.

To achieve the above object, the present invention proposes a method for calculating the coordinates of an aircraft door based on a binocular image, comprising:

Acquire a binocular image; input the binocular image into a first neural network model for first feature extraction, and obtain a hatch area sub-image through boundary regression; input the hatch area sub-image into a second neural network model for second feature extraction, and perform logistic regression on each pixel point of the hatch area sub-image to obtain a hatch seam edge map; extract the coordinates of the edge points based on the hatch seam edge map, and calculate two two-dimensional coordinate points of the hatch; and obtain two three-dimensional coordinate points of the hatch through triangulation calculation based on the two two-dimensional coordinate points.

The aircraft door coordinate calculation method based on binocular images in the embodiment of the present invention can accurately calculate the door coordinates in multiple dimensions, provide more sufficient and accurate door position information for the boarding bridge guidance system, fundamentally improve the docking accuracy of the bridge guidance system, and effectively solve the problems of high labor cost and frequent accidents in the traditional bridge docking method, which can play a role in the intelligence and automation of airports. The present invention has great theoretical and practical value.

In addition, the aircraft door coordinate calculation method based on binocular images according to the above embodiment of the present invention may also have the following additional technical features:

Furthermore, in one embodiment of the present invention, a training image library is constructed to obtain the first neural network model through training; and a training database is constructed to obtain the second neural network model through training.

Furthermore, in one embodiment of the present invention, after obtaining the door seam edge map, it also includes: obtaining the door seam edge points of the door seam edge map by edge point sampling; dividing the edge points into four categories of top, bottom, left and right by edge point classification, and obtaining four curve equations by calculating a weighted least squares module; and using the four curve equations to construct a global curve fitting error.

Furthermore, in one embodiment of the present invention, the coordinates of edge points are extracted based on the hatch door seam edge map to calculate two two-dimensional coordinate points of the hatch door, including: based on the hatch door seam edge map, using the edge point classification to respectively extract a left edge point set, a right edge point set and a lower edge point set; using ransac to fit the left edge point set and the right edge point set into two quadratic curves, and to fit the lower edge point set into a straight line; and respectively calculating the intersection points of the two quadratic curves with the straight line to obtain the two two-dimensional coordinate points of the hatch door.

Further, in one embodiment of the present invention, the binocular image includes a first binocular image and a second binocular image, and the two three-dimensional coordinate points of the hatch are obtained by triangulation based on the two two-dimensional coordinate points, including: the two two-dimensional coordinate points of the first binocular image obtained by preset calculation are respectively The two two-dimensional coordinate points of the second binocular image are The focal length of the stereo camera is preset to be f, the baseline is B, and the depth values of the two coordinate points of the first stereo image and the second stereo image are calculated respectively as follows:

The internal parameter matrix of the binocular camera is preset to be K, and the three-dimensional coordinates of two points in the first binocular image and the second binocular image are calculated as follows:

Furthermore, in one embodiment of the present invention, the method further includes: scaling and cropping the acquired binocular image, and inputting the image into the first neural network model.

Furthermore, in one embodiment of the present invention, two intersection points formed by the extension lines of the left and right edges of the hatch and the extension line of the lower edge of the hatch are used as two two-dimensional coordinate points of the hatch.

To achieve the above object, the present invention proposes, on the other hand, a door coordinate calculation device based on binocular images, comprising:

An acquisition module is used to acquire binocular images; a first extraction module is used to input the binocular images into a first neural network model for first feature extraction, and obtain a hatch area sub-image through boundary regression; a second extraction module is used to input the hatch area sub-image into a second neural network model for second feature extraction, and perform logistic regression on each pixel point of the hatch area sub-image to obtain a hatch door seam edge map; a first calculation module is used to extract the coordinates of edge points based on the hatch door seam edge map, and calculate two two-dimensional coordinate points of the hatch; a second calculation module is used to obtain two three-dimensional coordinate points of the hatch by triangulation calculation based on the two two-dimensional coordinate points.

The door coordinate calculation device based on binocular images in the embodiment of the present invention can accurately calculate the door coordinates in multiple dimensions, provide more sufficient and accurate door position information for the boarding bridge guidance system, fundamentally improve the docking accuracy of the bridge guidance system, effectively solve the problems of high labor cost and frequent accidents in the traditional bridge docking method, and can play a role in the intelligence and automation of the airport. The present invention has great theoretical and practical value.

Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

FIG1 is a schematic diagram of a docking scenario of a boarding bridge in a real scenario;

2 is a flow chart of a method for calculating door coordinates based on binocular images according to an embodiment of the present invention;

FIG3 is a schematic diagram of a picture captured by a camera according to an embodiment of the present invention;

FIG4 is a door area sub-image extracted according to an embodiment of the present invention;

FIG5 is a schematic diagram of a hatch door seam edge according to an embodiment of the present invention;

FIG6 is a schematic diagram of a hatch door coordinate point detection edge according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of the structure of a device for calculating hatch coordinates based on binocular images according to an embodiment of the present invention.

Detailed ways

It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

The following describes a method and device for calculating door coordinates based on binocular images according to an embodiment of the present invention with reference to the accompanying drawings.

FIG. 2 is a flow chart of a method for calculating door coordinates based on binocular images according to an embodiment of the present invention.

As shown in FIG2 , the door coordinate calculation method based on binocular images includes:

S1, collect binocular images.

As an example, the present invention installs a binocular camera at the front end of the corridor to collect images. Specifically, the aircraft door coordinate calculation method proposed in the present invention relies on the aircraft door image collected by the camera, and since the image is two-dimensional, in order to obtain the three-dimensional door coordinates, it is necessary to use a binocular camera (i.e., a camera with two cameras) to collect images. The image collected by the camera is shown in Figure 3.

The significance of this step is to collect binocular images so that the hatch coordinates can be further calculated.

S2, inputting the binocular image into the first neural network model to extract the first feature, and obtaining the hatch area sub-image through boundary regression.

It can be understood that this step is to extract the door area sub-image based on the depth feature. Since the camera fixed on the bridge is constantly moving, the content contained in the pictures it takes is also ever-changing. When the bridge is closer to the aircraft, the picture is almost entirely the aircraft door. When the bridge is farther away from the aircraft, the picture will contain other content such as the aircraft body and the airport environment. In order to minimize the impact of other content in the picture on the calculation of the door coordinates, it is necessary to first extract the door area in the picture.

The present invention first constructs a training image library and trains the neural network model offline. During actual operation, the images captured by the camera are scaled and cropped, and then sent to the neural network model for feature extraction. Finally, the accurate hatch area is obtained through boundary regression. The extracted hatch area sub-image is shown in Figure 4.

The purpose of this step is to extract the sub-image of the hatch area, thereby reducing the influence of other contents in the image captured by the camera on the calculation of the hatch coordinates.

S3, input the hatch area sub-image into the second neural network model to extract the second feature, and perform logistic regression on each pixel point of the hatch area sub-image to obtain the hatch door seam edge image.

It can be understood that this step is based on the detection of the hatch door seam edge under global curve constraints. The hatch door coordinates proposed in the present invention are calculated from the hatch door seam edge, so it is necessary to detect the hatch door seam edge in the image.

Specifically, the present invention first constructs a training database and trains the neural network model offline. Traditional methods usually directly use real edge images as supervisory signals to train the neural network model. However, compared with other types of edges, the edge of the hatch door seam is a distorted rectangle, the upper and lower edges are straight lines, and the left and right edges are quadratic curves. Therefore, the global curve feature of the hatch door seam edge can be used to improve the effect of edge detection.

The present invention proposes to add an additional global curve constraint module to further optimize the neural network model while using the edge map for supervision. After obtaining the edge map, firstly, an edge point sampling module is used to obtain uniformly distributed hatch door seam edge points, and then the edge point classification module is used to divide the collected edge points into four types: upper, lower, left, and right. Then, the equations of the four curves of upper, lower, left, and right are calculated by the weighted least squares module. Finally, the four curve equations are used to construct the global curve fitting error, and the performance of the edge extraction network is further optimized by back propagation.

During actual operation, the extracted hatch area sub-image is fed into the neural network model to extract features, and a logistic regression is performed on each pixel of the entire image to obtain the precise hatch door gap edge, as shown in FIG5 .

The purpose of this step is to detect the edge of the hatch door gap in the image for subsequent coordinate point calculation. At the same time, an optimization method based on global curve constraints is proposed to improve the accuracy of edge detection.

S4, extracting the coordinates of the edge points based on the hatch door seam edge map, and calculating two two-dimensional coordinate points of the hatch door.

It is understandable that this step is a two-dimensional coordinate point calculation based on multi-curve fitting. Specifically, when docking the boarding bridge, the floor of the bridge needs to be docked near the door pedal. Therefore, the present invention uses the two intersection points formed by the extension lines of the left and right edges of the door and the extension line of the lower edge of the door as the coordinate points of the door.

After obtaining the edge of the hatch door seam, the coordinates of each edge point can be further extracted. The left edge point set, right edge point set and lower edge point set are extracted respectively using the above-mentioned edge point classification module. Then, the left and right edge point sets are fitted into two quadratic curves using the RANSAC method, and the lower edge point set is fitted into a straight line. The intersection points of the two quadratic curves and the straight line are calculated respectively to obtain the two coordinate points of the hatch door, as shown in Figure 6.

The purpose of this step is to calculate the two-dimensional coordinates of the hatch door coordinate point from the image, in preparation for the subsequent calculation of the three-dimensional coordinates.

S5, based on the two two-dimensional coordinate points, obtain two three-dimensional coordinate points of the hatch through triangulation calculation.

Specifically, through the above steps, the two coordinate points corresponding to the two binocular images can be calculated respectively, and the three-dimensional coordinate points of the hatch can be obtained by triangulation. Assume that the two coordinate points calculated in the left figure of Figure 5 are The two coordinate points in the right figure are Assuming the focal length of the binocular camera is f and the baseline is B, the depth values of the two coordinate points can be calculated as follows:

Assuming that the intrinsic parameter matrix of the camera is K, the three-dimensional coordinates of the two points can be further calculated as:

Therefore, the purpose of this step is to convert the two-dimensional coordinates of the door coordinate point calculated in the image into three-dimensional coordinates relative to the corridor bridge, so as to guide the corridor bridge to dock with the aircraft door.

According to the door coordinate calculation method based on binocular images of the embodiment of the present invention, the three-dimensional coordinates of the aircraft door relative to the boarding bridge can be calculated in real time and accurately, so as to accurately guide the boarding bridge to dock with the aircraft door. The present invention uses a binocular camera to collect images, trains a neural network to extract the door area sub-graph, and uses the characteristic that the edge of the door seam can be approximated as four curves to realize a door seam edge detection method based on global curve constraints, and further fits the edge points into multiple curves, calculates the intersection of the curves, and uses the triangulation method to calculate the three-dimensional point coordinates of the door. Compared with other methods, the present invention can accurately calculate the door coordinates in 6 dimensions, provide more sufficient and accurate door position information for the boarding bridge guidance system, fundamentally improve the docking accuracy of the bridge guidance system, effectively solve the problems of high labor cost and frequent accidents of the traditional bridge docking method, and can play a role in the intelligence and automation of the airport. Therefore, the present invention has great theoretical and practical value.

In order to implement the above embodiment, as shown in Figure 7, a cabin door coordinate calculation device 10 based on binocular images is also provided in this embodiment. The device 10 includes: a collection module 100, a first extraction module 200, a second extraction module 300, a first calculation module 400 and a second calculation module 500.

The acquisition module 100 is used to acquire binocular images;

A first extraction module 200, configured to input the binocular image into a first neural network model to perform first feature extraction, and obtain a hatch area sub-image by boundary regression;

The second extraction module 300 is used to input the hatch area sub-image into the second neural network model to perform second feature extraction, and perform logistic regression on each pixel point of the hatch area sub-image to obtain a hatch door seam edge image;

A first calculation module 400 is used to extract the coordinates of edge points based on the hatch door seam edge map, and calculate two two-dimensional coordinate points of the hatch door;

The second calculation module 500 is used to obtain two three-dimensional coordinate points of the hatch through triangulation calculation based on the two two-dimensional coordinate points.

Furthermore, it also includes: a first training module, used to build a training image library, and train to obtain a first neural network model; a second training module, used to build a training database, and train to obtain a second neural network model.

Furthermore, the second extraction module 300 further includes:

A sampling module, used for obtaining hatch door seam edge points of a hatch door seam edge map by edge point sampling;

The classification module is used to classify edge points into four categories: up, down, left, and right, and calculate four curve equations through the weighted least squares module;

A fitting module is used to construct a global curve fitting error using four curve equations.

According to the binocular image-based door coordinate calculation device proposed in the embodiment of the present invention, the three-dimensional coordinates of the aircraft door relative to the boarding bridge can be calculated in real time and accurately, so as to accurately guide the boarding bridge to dock with the aircraft door. The present invention uses a binocular camera to collect images, trains a neural network to extract the door area sub-graph, utilizes the characteristic that the edge of the door seam can be approximated as four curves, and further fits the edge points into multiple curves, calculates the intersection of the curves and uses the triangulation method to calculate the three-dimensional point coordinates of the door. The present invention can accurately calculate the door coordinates in 6 dimensions, provide more sufficient and accurate door position information for the boarding bridge guidance system, fundamentally improve the docking accuracy of the bridge guidance system, effectively solve the problems of high labor cost and frequent accidents of the traditional bridge docking method, and can play a role in the intelligence and automation of the airport. Therefore, the present invention has great theoretical and practical value.

It should be noted that the above explanation of the embodiment of the method for calculating the cabin door coordinates based on binocular images is also applicable to the device for calculating the cabin door coordinates based on binocular images of this embodiment, and will not be repeated here.

In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims (9)

1. A method for calculating the door coordinates based on binocular images, comprising the following steps: Collect binocular images; Inputting the binocular image into a first neural network model to extract a first feature, and obtaining a hatch area sub-image by boundary regression; Inputting the hatch area sub-image into a second neural network model to perform second feature extraction, and performing logistic regression on each pixel point of the hatch area sub-image to obtain a hatch door seam edge image; Extracting the coordinates of edge points based on the hatch door gap edge map, and calculating two two-dimensional coordinate points of the hatch door; Based on the two two-dimensional coordinate points, two three-dimensional coordinate points of the hatch are obtained by triangulation calculation; The binocular image includes a first binocular image and a second binocular image, and the two three-dimensional coordinate points of the hatch are obtained by triangulation based on the two two-dimensional coordinate points, including: The two two-dimensional coordinate points of the first binocular image obtained by preset calculation are , , the two two-dimensional coordinate points of the second binocular image are , , the preset focal length of the binocular camera is , the baseline is , respectively calculate the depth values of the two coordinate points of the first binocular image and the second binocular image: The internal parameter matrix of the binocular camera is preset as , respectively calculate the three-dimensional coordinates of two points in the first binocular image and the second binocular image as: . 2. The method according to claim 1 is characterized in that a training image library is constructed to obtain the first neural network model through training; and a training database is constructed to obtain the second neural network model through training. 3. The method according to claim 1, characterized in that after obtaining the hatch door gap edge map, it also includes: Obtaining hatch door seam edge points of the hatch door seam edge map by edge point sampling; By classifying edge points, the edge points are divided into four categories: up, down, left, and right, and four curve equations are obtained by calculating the least squares module with weights; A global curve fitting error is constructed using the four curve equations. 4. The method according to claim 3, characterized in that the step of extracting the coordinates of the edge points based on the hatch door seam edge map and calculating the two two-dimensional coordinate points of the hatch door comprises: Based on the hatch door seam edge map, using the edge point classification to respectively extract a left edge point set, a right edge point set and a lower edge point set; Use ransac to fit the left edge point set and the right edge point set into two quadratic curves, and fit the lower edge point set into a straight line; The intersection points of the two quadratic curves and the straight line are calculated respectively to obtain two two-dimensional coordinate points of the hatch. 5. The method according to claim 1 is characterized in that the method also includes: scaling and cropping the acquired binocular image and inputting it into the first neural network model. 6. The method according to claim 1 is characterized in that two intersection points formed by the extension lines of the left edge and the right edge of the hatch and the extension line of the lower edge of the hatch are used as the two two-dimensional coordinate points of the hatch. 7. A door coordinate calculation device based on binocular images, comprising: An acquisition module, used for acquiring binocular images; A first extraction module, used for inputting the binocular image into a first neural network model to perform first feature extraction, and obtaining a hatch area sub-image by boundary regression; A second extraction module is used to input the hatch area sub-image into a second neural network model to perform second feature extraction, and perform logistic regression on each pixel point of the hatch area sub-image to obtain a hatch door seam edge image; A first calculation module, used for extracting the coordinates of edge points based on the hatch door seam edge map, and calculating two two-dimensional coordinate points of the hatch door; A second calculation module, used for obtaining two three-dimensional coordinate points of the hatch through triangulation calculation based on the two two-dimensional coordinate points; The second calculation module is also used to preset the two two-dimensional coordinate points of the first binocular image obtained by calculation as follows: , , the two 2D coordinate points of the second binocular image are , , the preset focal length of the binocular camera is , the baseline is , respectively calculate the depth values of the two coordinate points of the first binocular image and the second binocular image: The internal parameter matrix of the binocular camera is preset as , respectively calculate the three-dimensional coordinates of two points in the first binocular image and the second binocular image as: . 8. The device according to claim 7 is characterized in that it also includes: a first training module for constructing a training image library and training to obtain the first neural network model; a second training module for constructing a training database and training to obtain the second neural network model. 9. The device according to claim 7, wherein the second extraction module further comprises: A sampling module, used for obtaining the hatch door seam edge points of the hatch door seam edge map by edge point sampling; A classification module is used to classify edge points into four categories: upper, lower, left, and right, and obtain four curve equations by calculating with a weighted least squares module; A fitting module is used to construct a global curve fitting error using the four curve equations.