CN114863385B - Road curved surface information generation method, device, equipment and computer readable medium - Google Patents

Abstract

The embodiment of the disclosure discloses a road curved surface information generation method, a road curved surface information generation device, equipment and a computer readable medium. One embodiment of the method comprises: extracting key points of each road image in the pre-acquired road image sequence to generate an obstacle key point coordinate sequence set, and obtaining an obstacle key point coordinate sequence set; constructing a right-angle constraint equation based on the set of the coordinate series groups of the key points of the obstacles; updating the initial road surface equation based on the right-angle constraint equation to obtain a target road surface equation; and determining the target road surface equation as the road surface information. This embodiment may improve the accuracy of the generated road surface information.

Description

Road surface information generation method, device, equipment and computer readable medium

Technical Field

Embodiments of the present disclosure relate to the field of computer technology, and in particular to methods, devices, equipment, and computer-readable media for generating road surface information.

Background Art

The generation of road surface information is of great significance to the field of autonomous driving. At present, the commonly used method to generate road surface information is to extract static features (such as lane lines) from road images, obtain the three-dimensional coordinates of the features through triangulation, and then generate road surface information.

However, when the above method is used to generate road surface information, the following technical problems often occur:

First, the static features of the road surface are easily blocked, resulting in inaccurate extraction of static features, which reduces the accuracy of the generated road surface information.

Second, the static features of the road surface lack obvious feature points, which results in the extraction of static features being inaccurate, thereby reducing the accuracy of the generated road surface information.

Summary of the invention

The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

Some embodiments of the present disclosure propose methods, devices, equipment and computer-readable media for generating road surface information to solve one or more of the technical problems mentioned in the above background technology section.

In a first aspect, some embodiments of the present disclosure provide a method for generating road surface information, the method comprising: extracting key points from each road image in a pre-acquired road image sequence to generate an obstacle key point coordinate sequence group, and obtaining an obstacle key point coordinate sequence group set; constructing a rectangular constraint equation based on the above obstacle key point coordinate sequence group set; updating an initial road surface equation based on the above rectangular constraint equation to obtain a target road surface equation; and determining the above target road surface equation as the road surface information.

In a second aspect, some embodiments of the present disclosure provide a road surface information generating device, which includes: an extraction unit, configured to perform key point extraction on each road image in a pre-acquired road image sequence to generate an obstacle key point coordinate sequence group, and obtain an obstacle key point coordinate sequence group set; a construction unit, configured to construct a rectangular constraint equation based on the above-mentioned obstacle key point coordinate sequence group set; an updating unit, configured to update the initial road surface equation based on the above-mentioned rectangular constraint equation to obtain a target road surface equation; and a determination unit, configured to determine the above-mentioned target road surface equation as the road surface information.

In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the road surface information generation method of some embodiments of the present disclosure, the accuracy of the generated road surface information can be improved. Specifically, the reason for the reduced accuracy of the generated road surface information is that the static features of the road surface are easily blocked, resulting in the extracted static features being inaccurate. Based on this, the road surface information generation method of some embodiments of the present disclosure first extracts key points from each road image in the pre-acquired road image sequence to generate an obstacle key point coordinate sequence group, and obtains an obstacle key point coordinate sequence group set. By extracting the obstacle key point coordinates, the road surface information can be generated without relying on static features. It avoids the situation that the static features of the road surface are easily blocked, resulting in the extracted static features being inaccurate. Then, based on the above obstacle key point coordinate sequence group set, a rectangular constraint equation is constructed. By constructing the rectangular constraint equation, it can be used to improve the accuracy of the generated road surface information. Afterwards, based on the above rectangular constraint equation, the initial road surface equation is updated to obtain the target road surface equation. By updating, the accuracy of the road surface equation can be further improved. Finally, the target road surface equation is determined as the road surface information. Therefore, the road surface information generation method of some embodiments of the present disclosure constructs a rectangular constraint equation based on the coordinates of the key points of the obstacle, which can improve the accuracy of the generated road surface information.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

FIG1 is a flow chart of some embodiments of a method for generating road surface information according to the present disclosure;

FIG2 is a flow chart of other embodiments of the method for generating road surface information according to the present disclosure;

FIG3 is a schematic diagram of the structure of some embodiments of the road surface information generating device according to the present disclosure;

FIG. 4 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

FIG1 shows a process 100 of some embodiments of the method for generating road surface information according to the present disclosure. The process 100 of the method for generating road surface information comprises the following steps:

Step 101 : extract key points from each road image in a pre-acquired road image sequence to generate an obstacle key point coordinate sequence group, thereby obtaining an obstacle key point coordinate sequence group set.

In some embodiments, the execution body of the road surface information generation method may extract key points from each road image in the pre-acquired road image sequence to obtain a set of obstacle key point coordinate sequence groups. Wherein, each road image in the road image sequence may be a continuous frame image taken by a vehicle-mounted camera. Key points may be extracted from each road image in the pre-acquired road image sequence by a preset extraction algorithm to obtain a set of obstacle key point coordinate sequence groups. Each obstacle key point coordinate sequence may correspond to an obstacle. Each obstacle key point coordinate sequence group may correspond to each obstacle in a road image. The obstacle key point may represent the outermost point of the tire of the obstacle vehicle in contact with the ground. The coordinates of each obstacle key point in the obstacle key point coordinate sequence may be arranged in a preset order. In addition, the number of detection key point coordinates in the detection key point coordinate sequence may be: one, two, three or four, etc. The above preset order may be a clockwise order or a counterclockwise order, for example, a counterclockwise order: the order of right front wheel, left front wheel, left rear wheel, and right rear wheel.

As an example, the extraction algorithm may include, but is not limited to, at least one of the following: GUP (GeometryUncertainty Projection, monocular three-dimensional target detection network), SegNet (image semantic segmentation deep network), FCN (Fully Convolutional Networks, full convolution neural network) model, VGG (Visual Geometry GroupNetwork, convolutional neural network) model or GoogLeNet (deep neural network) model, etc. For example, three points corresponding to the right front wheel, left front wheel, and left rear wheel of the obstacle vehicle are detected. The order of the corresponding detection key point coordinates can be sorted counterclockwise starting from the right front wheel. Then, the order numbering of the obstacle key point coordinates corresponding to the right front wheel, left front wheel, and left rear wheel can be (1-2-3).

Step 102: construct a rectangular constraint equation based on the obstacle key point coordinate sequence group set.

In some embodiments, the execution entity may construct a rectangular constraint equation in various ways based on the obstacle key point coordinate sequence group set.

In some optional implementations of some embodiments, the execution subject constructs a rectangular constraint equation based on the obstacle key point coordinate sequence group set, which may include the following steps:

In the first step, each obstacle key point coordinate sequence in the above obstacle key point coordinate sequence group set is screened to obtain a target obstacle key point coordinate sequence group set.

In some optional implementations of some embodiments, the execution subject screens each obstacle key point coordinate sequence in the obstacle key point coordinate sequence group set to obtain a target obstacle key point coordinate sequence group set, which may include the following steps:

For each obstacle key point coordinate sequence in the above obstacle key point coordinate sequence group set, perform the following screening processing steps:

In the first sub-step, each obstacle key point coordinate in the above obstacle key point coordinate sequence is back-projected to obtain a three-dimensional obstacle key point coordinate sequence. The obstacle key point coordinates can be back-projected from the image coordinate system to the vehicle coordinate system by an inverse perspective transformation method. In this way, a three-dimensional obstacle key point coordinate sequence can be obtained.

The second sub-step is to construct a unit vector sequence using the coordinates of each three-dimensional obstacle key point in the three-dimensional obstacle key point coordinate sequence. For every two adjacent three-dimensional obstacle key point coordinates, a unit vector can be constructed. The unit vector group can be generated by the following formula:

Wherein, j and p represent sequence numbers. l represents a unit vector in the above unit vector sequence. _{l j} represents the jth unit vector in the above unit vector sequence. w represents the coordinates of the three-dimensional obstacle key point in the above three-dimensional obstacle key point coordinate sequence. _{w p} represents the coordinates of the pth three-dimensional obstacle key point in the above three-dimensional obstacle key point coordinate sequence. _{w p+1} represents the coordinates of the p+1th three-dimensional obstacle key point in the above three-dimensional obstacle key point coordinate sequence. ||·|| ₂ represents the 2-normal form.

The third sub-step is to determine the three-dimensional obstacle key point coordinate sequence as the target obstacle key point coordinate sequence in response to determining that the coordinates of each obstacle key point in the obstacle key point coordinate sequence meet the preset key point condition, and the unit vector sequence meets the preset vector relationship condition. The preset key point condition may be: the number of obstacle key points in the obstacle key point coordinate sequence is a preset number (for example, 3), and the sequence number of each obstacle key point is a sequence number in a preset sequence number set. The preset vector relationship condition may be that every two adjacent unit vectors in the unit vector sequence are in a vertical relationship. The order of each unit vector in the unit vector sequence may correspond to the order in the obstacle key point coordinate sequence.

As an example, the above-mentioned sequential numbering set can be: {(1-2-3), (2-3-4), (1-4-3), (2-1-4)}. For example, the sequential numbering of each obstacle key point is (1-2-3). Then it can be determined that the above-mentioned obstacle key point coordinate sequence meets the above-mentioned preset key point conditions. Thus, there can be two unit vectors generated, namely: the unit vector formed by the coordinates of the three-dimensional obstacle key points of sequential numbers 1 and 2 through the above formula, and the unit vector formed by the coordinates of the three-dimensional obstacle key points of sequential numbers 2 and 3 through the above formula. If the two unit vectors are perpendicular, it can be determined that the unit vector sequence meets the preset vector relationship conditions.

In practice, the sequential numbering of the coordinates of the key points of obstacles can not only be used to distinguish the wheels of the corresponding obstacle vehicles, but also facilitate the generation of unit vectors. Also because of the introduction of sequential numbering, the generated unit vector can represent the boundary of the rectangle that approximates the vehicle from a bird's-eye view. By introducing the preset key point conditions, the obstacle key point coordinate sequence that meets the preset key point conditions can be screened out for constructing the right-angle constraint equation. In addition, by introducing the preset vector relationship conditions, it can be used to determine that the unit vector is the boundary of the rectangular frame and is perpendicular to each other. In this way, the accuracy of representing the actual obstacle vehicle is improved. Furthermore, it can be used to improve the accuracy of the generated road surface information.

The second step is to construct a rectangular constraint equation based on the above target obstacle key point coordinate sequence set. Among them, the unit vector sequence corresponds one-to-one with the target obstacle key point coordinate sequence in the target obstacle key point coordinate sequence set. The constructed rectangular constraint equation can be:

表示与上述单位向量序列中的第j个单位向量对应的直角关系误差。l_j表示上述单位向量序列中的第j个单位向量。l_j+1表示与上述单位向量序列中的第j+1个单位向量。Among them, e ¹ represents the result of the rectangular constraint equation, that is, the rectangular relationship error.

represents the rectangular relationship error corresponding to the jth unit vector in the above unit vector sequence. _{l j} represents the jth unit vector in the above unit vector sequence. _{l j+1} represents the j+1th unit vector in the above unit vector sequence.

Step 103: Based on the rectangular constraint equation, the initial road surface equation is updated to obtain the target road surface equation.

In some embodiments, the execution entity may update the initial road surface equation in various ways based on the rectangular constraint equation to obtain the target road surface equation.

In some optional implementations of some embodiments, the initial road surface equation is generated by the following steps:

The first step is to extract lane line points from a preset target road image to obtain a set of lane line point coordinates. The target road image may be a first frame of a road image acquired in advance. The lane line points may be extracted from the preset target road image using the above extraction algorithm to obtain a set of lane line point coordinates.

The second step is to back-project the coordinates of each lane point in the above lane point coordinate set to the vehicle coordinate system to obtain a three-dimensional coordinate set of lane points. The three-dimensional coordinate set of lane points can be obtained by back-projecting the coordinates of each lane point in the above lane point coordinate set from the image coordinate system to the vehicle coordinate system through an inverse perspective transformation.

The third step is to generate the initial road surface equation based on the lane line point three-dimensional coordinate set. The three-dimensional coordinates of the lane line points in the lane line point three-dimensional coordinate set can be input into the preset surface equation to solve the parameters of the initial road surface equation. The initial road surface equation can be obtained. The expression of the initial surface equation can be as follows:

Wherein, P(s) represents the initial surface equation. s represents the three-dimensional coordinates of the lane line point in the above lane line point three-dimensional coordinate set. x represents the horizontal coordinate value of the lane line point three-dimensional coordinate. y represents the vertical coordinate value of the lane line point three-dimensional coordinate. z represents the vertical coordinate value of the lane line point three-dimensional coordinate. A represents the coefficient matrix of the initial surface equation. B represents the coefficient vector of the initial surface equation. c represents the constant term. _{a 1} , a ₂ , a ₃ represent the data in the coefficient matrix. _{b 1} , b ₂ represent the data in the coefficient vector. T represents the transpose of the matrix.

Specifically, the coefficient matrix in the initial surface equation may be a zero matrix.

Step 104: determine the target road surface equation as road surface information.

In some embodiments, the execution entity may determine the target road surface equation as road surface information, wherein the road surface information may be information characterizing the road surface.

Optionally, the above execution entity may further perform the following steps:

The first step is to extract feature points from each road image in the road image sequence to obtain a road feature point coordinate set. The extraction algorithm can be used to extract feature points from each road image in the road image sequence to obtain a road feature point coordinate set. The coordinates of each road feature point in the road feature point coordinate set can be used to characterize the lane line corresponding to the road image.

The second step is to back-project the coordinates of each road feature point in the above-mentioned road feature point coordinate set to the coordinate system where the above-mentioned target road surface equation is located, and obtain the back-projected feature point coordinate set. Among them, the coordinates of each road feature point in the above-mentioned road feature point coordinate set can be back-projected from the image coordinate system to the coordinate system where the above-mentioned target road surface equation is located by the method of inverse projection transformation to obtain the back-projected feature point coordinate set. The coordinate system where the target road surface equation is located can be a vehicle coordinate system. On the basis of improving the accuracy of the target road surface equation, by back-projecting the above-mentioned road feature point coordinates to the coordinate system where the above-mentioned target road surface equation is located, the accuracy of the generated back-projected feature point coordinate set can be improved.

The third step is to send the above-mentioned back-projected feature point coordinate set and the above-mentioned road surface information to a display terminal for display. Among them, since the accuracy of the above-mentioned back-projected feature point coordinate set and the above-mentioned road surface information is improved, the accuracy of the road information displayed by the display terminal can be improved. Furthermore, it can be used to improve driving safety.

The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the road surface information generation method of some embodiments of the present disclosure, the accuracy of the generated road surface information can be improved. Specifically, the reason for the reduced accuracy of the generated road surface information is that the static features of the road surface are easily blocked, resulting in the extracted static features being inaccurate. Based on this, the road surface information generation method of some embodiments of the present disclosure first extracts key points from each road image in the pre-acquired road image sequence to obtain a set of obstacle key point coordinate sequence groups. By extracting the coordinates of the obstacle key points, the road surface information can be generated without relying on static features. It avoids the situation that the static features of the road surface are easily blocked, resulting in the extracted static features being inaccurate. Then, based on the above obstacle key point coordinate sequence group set, a rectangular constraint equation is constructed. By constructing the rectangular constraint equation, it can be used to improve the accuracy of the generated road surface information. Afterwards, based on the above rectangular constraint equation, the initial road surface equation is updated to obtain the target road surface equation. By updating, the accuracy of the road surface equation can be further improved. Finally, the above target road surface equation is determined as the road surface information. Therefore, the road surface information generation method of some embodiments of the present disclosure constructs a rectangular constraint equation based on the coordinates of the key points of the obstacle, which can improve the accuracy of the generated road surface information.

Further referring to FIG. 2 , it shows a process 200 of another embodiment of a method for generating road surface information. The process 200 of the method for generating road surface information comprises the following steps:

Step 201 : extract key points from each road image in the pre-acquired road image sequence to generate an obstacle key point coordinate sequence group, thereby obtaining an obstacle key point coordinate sequence group set.

Step 202: construct a rectangular constraint equation based on the obstacle key point coordinate sequence group set.

In some embodiments, the specific implementation methods and technical effects of steps 201-202 can refer to steps 101-102 in the embodiments corresponding to FIG. 1, and will not be repeated here.

Step 203: Generate a key point projection error sequence group set based on a preset camera intrinsic parameter matrix and a coordinate transformation matrix.

In some embodiments, the execution subject may generate a key point projection error sequence group set based on a preset camera intrinsic parameter matrix and a coordinate transformation matrix. The key point projection error sequence group set may be generated by the following formula:

表示关键点投影误差序列组集合中第i个关键点投影误差序列组中的关键点投影误差。

表示关键点投影误差序列组集合中第i个关键点投影误差序列组中第k个关键点投影误差序列中的关键点投影误差。

表示关键点投影误差序列组集合中第i个关键点投影误差序列组中第k个关键点投影误差序列中的第n个关键点投影误差。K表示相机内参矩阵。R表示坐标旋转矩阵。m表示上述目标障碍物关键点坐标序列组集合中的目标障碍物关键点坐标。m_i表示上述目标障碍物关键点坐标序列组集合中第i个目标障碍物关键点坐标序列组中的目标障碍物关键点坐标。m_i，p表示上述目标障碍物关键点坐标序列组集合中第i个目标障碍物关键点坐标序列组中第p个目标障碍物关键点坐标序列中的目标障碍物关键点坐标。m_i，p，n表示上述目标障碍物关键点坐标序列组集合中第i个目标障碍物关键点坐标序列组中第p个目标障碍物关键点坐标序列中的第n个目标障碍物关键点坐标。k表示与上述目标障碍物关键点坐标序列组集合中的目标障碍物关键点坐标对应的障碍物关键点坐标。k_i表示与上述目标障碍物关键点坐标序列组集合中第i个目标障碍物关键点坐标序列组中的目标障碍物关键点坐标对应的障碍物关键点坐标。k_i，p表示与上述目标障碍物关键点坐标序列组集合中第i个目标障碍物关键点坐标序列组中第p个目标障碍物关键点坐标序列中的目标障碍物关键点坐标对应的障碍物关键点坐标。k_i，p，n表示与上述目标障碍物关键点坐标序列组集合中第i个目标障碍物关键点坐标序列组中第p个目标障碍物关键点坐标序列中的第n个目标障碍物关键点坐标对应的障碍物关键点坐标。N表示正态分布符号。∑_p表示预设的投影误差协方差矩阵。()₃表示取括号内向量的第3个元素。()_1：2表示取括号内向量的第1个到第2个元素。Where i, k, and n represent serial numbers. ^{e 2} represents the key point projection error in the key point projection error sequence group set.

Represents the key point projection error in the i-th key point projection error sequence group in the key point projection error sequence group set.

Represents the key point projection error in the kth key point projection error sequence in the ith key point projection error sequence group in the key point projection error sequence group set.

represents the nth key point projection error in the kth key point projection error sequence in the i-th key point projection error sequence group in the key point projection error sequence group set. K represents the camera intrinsic parameter matrix. R represents the coordinate rotation matrix. m represents the target obstacle key point coordinates in the above target obstacle key point coordinate sequence group set. _mi represents the target obstacle key point coordinates in the i-th target obstacle key point coordinate sequence group in the above target obstacle key point coordinate sequence group set. _mi,p represents the target obstacle key point coordinates in the p-th target obstacle key point coordinate sequence in the i-th target obstacle key point coordinate sequence group in the above target obstacle key point coordinate sequence group set. _mi,p,n represents the n-th target obstacle key point coordinates in the p-th target obstacle key point coordinate sequence in the i-th target obstacle key point coordinate sequence group in the above target obstacle key point coordinate sequence group set. k represents the obstacle key point coordinates corresponding to the target obstacle key point coordinates in the above target obstacle key point coordinate sequence group set. k _i represents the obstacle key point coordinates corresponding to the target obstacle key point coordinates in the i-th target obstacle key point coordinate sequence group in the above target obstacle key point coordinate sequence group set. _{k i,p} represents the obstacle key point coordinates corresponding to the target obstacle key point coordinates in the p-th target obstacle key point coordinate sequence in the i-th target obstacle key point coordinate sequence group in the above target obstacle key point coordinate sequence group set. _{k i,p,n} represents the obstacle key point coordinates corresponding to the n-th target obstacle key point coordinates in the p-th target obstacle key point coordinate sequence in the i-th target obstacle key point coordinate sequence group in the above target obstacle key point coordinate sequence group set. N represents the symbol of normal distribution. _{∑ p} represents the preset projection error covariance matrix. () ₃ represents taking the third element of the vector in the brackets. () _1:2 represents taking the first to second elements of the vector in the brackets.

Step 204: Based on the rectangular constraint equation, the initial road surface equation is updated to obtain the target road surface equation.

In some embodiments, the execution subject may update the initial road surface equation based on the key point projection error sequence group set, the rectangular constraint equation and the preset covariance matrix to obtain the target road surface equation. First, the initial state vector of the initial state equation may be determined. The initial state vector may be composed of the data in the coefficient matrix and the coefficient vector and the constant term of the initial surface equation. For example: Z = [c, b ₁ , b ₂ , a ₁ , a ₂ , a ₃ ]. Z may represent the initial state vector. Then, the target state vector sequence may be obtained by the following formula:

表示与目标障碍物关键点坐标序列组集合中第i个目标障碍物关键点坐标序列组中第p个目标障碍物关键点坐标序列中的第j个目标障碍物关键点坐标对应的约束方程的结果，即直角关系误差。

表示关键点投影误差序列组集合中第i个关键点投影误差序列组中第k个关键点投影误差序列中的第n个关键点投影误差的转置矩阵。

表示预设的投影误差协方差矩阵的逆矩阵。λ表示预设的投影参数组中的投影参数。表示预设的投影参数组中的第i个投影参数。D表示转换参数，用于缩短公式长度。Wherein, Z represents the above initial state vector. Z′ represents the target state vector in the target state vector sequence. Z′ _i represents the i-th target state vector in the target state vector sequence. P(m _{i, p, n} ) represents the result of inputting the n-th target obstacle key point coordinates in the p-th target obstacle key point coordinate sequence in the i-th target obstacle key point coordinate sequence group in the above target obstacle key point coordinate sequence group set into the initial state equation.

It represents the result of the constraint equation corresponding to the j-th target obstacle key point coordinate in the p-th target obstacle key point coordinate sequence in the i-th target obstacle key point coordinate sequence group in the target obstacle key point coordinate sequence group set, that is, the rectangular relationship error.

Represents the transposed matrix of the nth key point projection error in the kth key point projection error sequence in the ith key point projection error sequence group in the key point projection error sequence group set.

represents the inverse matrix of the preset projection error covariance matrix. λ represents the projection parameter in the preset projection parameter group. represents the i-th projection parameter in the preset projection parameter group. D represents the conversion parameter, which is used to shorten the formula length.

Finally, the last target state vector in the target state vector sequence can be determined as a parameter of the target road surface equation, thereby completing the update of the initial road surface equation to obtain the target road surface equation.

In practice, the projection parameter is related to the degree of bumping of the vehicle: the greater the degree of bumping, the lower the data credibility of the obstacle vehicle, and the smaller the projection parameter, so that the result of the projection parameter is less important. Therefore, it can be used to reduce the impact of the bumping degree on the generation of the target state vector. In this way, the accuracy of the generated target state vector is improved. In addition, the formula for generating the target state vector can be solved by a nonlinear optimization method. For example, the nonlinear optimization method may include but is not limited to at least one of the following: ISAM (Incremental Smoothing And Mapping), GTSAM (nonlinear optimization library), etc. In the solution process, it is necessary to meet the condition that the key point projection error satisfies the Gaussian distribution. The inverse of the covariance matrix characterizes the degree of certainty of the error. The larger the error value, the greater the degree of certainty and the smaller the uncertainty. Therefore, the inverse of the coordinate covariance matrix is introduced to reduce the impact of the error when generating the coordinates of the key points of the obstacle. Thus, various errors can be reduced and the accuracy of the generated target surface equation can be improved.

The above formulas and their related contents, as an inventive point of the embodiments of the present disclosure, solve the second technical problem mentioned in the background technology, "the static features of the road surface lack obvious feature points, which leads to the inaccuracy of the extracted static features, and then reduces the accuracy of the generated road surface information". The factors that lead to the reduction of the accuracy of the generated road surface information are often as follows: the static features of the road surface lack obvious feature points, which leads to the inaccuracy of the extracted static features. If the above factors are solved, the accuracy of the generated road surface information can be improved. In order to achieve this effect, first, the coordinates of the key points of the obstacles can be extracted to replace the static features of the road surface. Then, by generating the formula of the unit vector, the unit vector can be generated, so as to facilitate the construction of the constraint relationship. After that, the rectangular constraint equation can be used to generate the rectangular relationship error. It is used to improve the accuracy of generating the target state vector. Then, by introducing the initial surface equation, it is convenient to generate the initial state vector. This facilitates the generation of the target state vector. Then, by generating the formula of the key point projection error, the constraint conditions can be further increased to improve the accuracy of solving the target state vector. Finally, by generating the formula of the target state vector, it is possible to generate the target state vector when the above conditions are met. Thus, the accuracy of the target state vector is improved. Furthermore, it can be used to improve the accuracy of the road surface information.

Step 205: determine the target road surface equation as road surface information.

In some embodiments, the specific implementation of step 205 and the technical effects brought about can refer to step 104 in the embodiments corresponding to FIG. 1 , and will not be repeated here.

As can be seen from FIG. 2 , compared with the description of some embodiments corresponding to FIG. 1 , the process 200 of the road surface information generation method in some embodiments corresponding to FIG. 2 embodies the step of updating the initial road surface equation. Through the above formulas and their related contents, the technical problem of "the static features of the road surface lack obvious feature points, thereby resulting in the inaccuracy of the extracted static features" is solved. In turn, the accuracy of the generated road information is improved.

Further referring to FIG. 3 , as an implementation of the methods shown in the above-mentioned figures, the present disclosure provides some embodiments of a road surface information generating device, which correspond to the method embodiments shown in FIG. 2 , and the device can be specifically applied to various electronic devices.

As shown in FIG3 , the road surface information generating device 300 of some embodiments includes: an extracting unit 301, a constructing unit 302, an updating unit 303 and a determining unit 304. The extracting unit 301 is configured to extract key points from each road image in the pre-acquired road image sequence to generate an obstacle key point coordinate sequence group, and obtain an obstacle key point coordinate sequence group set; the constructing unit 302 is configured to construct a rectangular constraint equation based on the above obstacle key point coordinate sequence group set; the updating unit 303 is configured to update the initial road surface equation based on the above rectangular constraint equation to obtain a target road surface equation; the determining unit 304 is configured to determine the above target road surface equation as the road surface information.

It is understandable that the units recorded in the device 300 correspond to the steps in the method described with reference to Figure 1. Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units contained therein, and will not be repeated here.

Referring to Fig. 4, a schematic diagram of the structure of an electronic device 400 suitable for implementing some embodiments of the present disclosure is shown. The electronic device shown in Fig. 4 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG4 , the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to the bus 404.

Typically, the following devices can be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication device 409 can allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 4 shows an electronic device 400 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or provided alternatively. Each box shown in FIG. 4 may represent one device, or may represent multiple devices as needed.

In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

The computer-readable medium may be included in the device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: extracts key points from each road image in the pre-acquired road image sequence to generate an obstacle key point coordinate sequence group, and obtains an obstacle key point coordinate sequence group set; constructs a rectangular constraint equation based on the obstacle key point coordinate sequence group set; updates the initial road surface equation based on the rectangular constraint equation to obtain a target road surface equation; and determines the target road surface equation as the road surface information.

Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be provided in a processor, for example, may be described as: a processor including an extraction unit, a construction unit, an update unit, and a determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the extraction unit may also be described as a "unit for extracting a set of obstacle key point coordinate sequence groups".

The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims (5)

1. A road curved surface information generation method comprises the following steps:

extracting key points of each road image in the pre-acquired road image sequence to generate an obstacle key point coordinate sequence set, and obtaining an obstacle key point coordinate sequence set;

constructing a right-angle constraint equation based on the set of the coordinate series groups of the key points of the barrier;

updating the initial road surface equation based on the right-angle constraint equation to obtain a target road surface equation;

determining the target road surface equation as road surface information;

wherein, the constructing a right angle constraint equation based on the set of the coordinate series groups of the key points of the obstacles comprises:

screening each barrier key point coordinate sequence in the set of barrier key point coordinate sequences to obtain a set of target barrier key point coordinate sequences;

constructing a right-angle constraint equation based on the set of the target obstacle key point coordinate sequence group;

the screening processing is performed on each obstacle key point coordinate sequence in the set of obstacle key point coordinate sequences to obtain a set of target obstacle key point coordinate sequences, and the method includes:

for each obstacle key point coordinate sequence in the set of obstacle key point coordinate sequences, performing the following screening processing steps:

carrying out back projection on each barrier key point coordinate in the barrier key point coordinate sequence to obtain a three-dimensional barrier key point coordinate sequence;

constructing a unit vector sequence by utilizing each three-dimensional barrier key point coordinate in the three-dimensional barrier key point coordinate sequence;

in response to the fact that the coordinates of each barrier key point in the barrier key point coordinate sequence meet preset key point conditions and the unit vector sequence meets preset vector relation conditions, determining the three-dimensional barrier key point coordinate sequence as a target barrier key point coordinate sequence;

wherein the right angle constraint equation is constructed by the following steps:

wherein e is ¹ The result, which represents the right angle constraint equation, i.e. the right angle relationship error,

representing the rectangular error, l, corresponding to the jth unit vector in the sequence of unit vectors _j Represents the j unit vector, l, in the unit vector sequence _j+1 Represents the j +1 th unit vector in the unit vector sequence;

wherein the initial road surface equation is generated by:

extracting lane line points of a preset target road image to obtain a lane line point coordinate set;

back projecting each lane line point coordinate in the lane line point coordinate set to a vehicle coordinate system to obtain a lane line point three-dimensional coordinate set;

generating the initial road surface equation based on the three-dimensional coordinate set of the lane line points; before updating the initial road surface equation based on the right-angle constraint equation to obtain the target road surface equation, the method further comprises:

generating a set of key point projection error sequence groups based on a preset camera internal reference matrix and a coordinate transformation matrix;

updating the initial road surface equation based on the right-angle constraint equation to obtain a target road surface equation, wherein the method comprises the following steps:

updating an initial road surface equation based on the set of the key point projection error sequence groups, the right-angle constraint equation and a preset covariance matrix to obtain a target road surface equation;

updating an initial road surface equation based on the set of the key point projection error sequence groups, the right-angle constraint equation and a preset covariance matrix to obtain a target road surface equation, wherein the method comprises the following steps:

determining an initial state vector of an initial state equation;

obtaining a target state vector sequence by the following formula:

wherein Z represents the initial state vector, Z 'represents a target state vector in a target state vector sequence, Z' _i Representing the ith target state vector, P (m), in the sequence of target state vectors _i，p，n ) Representing the ith target obstacle key point coordinate sequence in the target obstacle key point coordinate sequence setThe nth target obstacle key point coordinate in the pth target obstacle key point coordinate sequence in the group is input to the result of the initial state equation,

representing the result of a constraint equation corresponding to the jth target obstacle key point coordinate in the jth target obstacle key point coordinate sequence group in the ith target obstacle key point coordinate sequence group in the target obstacle key point coordinate sequence group, namely the orthogonal relation error, is/are>

A transpose matrix representing the nth keypoint projection error in the kth keypoint projection error sequence in the ith keypoint projection error sequence set in the set of keypoint projection error sequences, device for combining or screening>

The method comprises the steps of representing an inverse matrix of a preset projection error covariance matrix, representing a projection parameter in a preset projection parameter group, representing an ith projection parameter in the preset projection parameter group, and representing a conversion parameter for shortening the length of a formula;

and determining the last target state vector in the target state vector sequence as a parameter of the target road surface equation to complete the updating of the initial road surface equation to obtain the target road surface equation.

2. The method of claim 1, wherein the method further comprises:

extracting characteristic points of each road image in the road image sequence to obtain a road surface characteristic point coordinate set;

carrying out back projection on the coordinates of each road surface characteristic point in the road surface characteristic point coordinate set to a coordinate system where the target road curved surface equation is located to obtain a back projection characteristic point coordinate set;

and sending the back projection feature point coordinate set and the road curved surface information to a display terminal for display.

3. A road surface information generating apparatus comprising:

the extraction unit is configured to extract key points of each road image in the pre-acquired road image sequence to generate an obstacle key point coordinate sequence group to obtain an obstacle key point coordinate sequence group set;

the building unit is configured to build a right angle constraint equation based on the set of the barrier key point coordinate series groups;

the updating unit is configured to update the initial road surface equation based on the right-angle constraint equation to obtain a target road surface equation;

a determination unit configured to determine the target road surface equation as road surface information;

wherein, the constructing a right angle constraint equation based on the set of the coordinate series groups of the key points of the obstacles comprises:

screening each barrier key point coordinate sequence in the set of barrier key point coordinate sequences to obtain a set of target barrier key point coordinate sequences;

constructing a right-angle constraint equation based on the set of the key point coordinate series of the target barrier;

the screening of each obstacle key point coordinate sequence in the set of obstacle key point coordinate sequences to obtain a set of target obstacle key point coordinate sequences includes:

for each obstacle key point coordinate sequence in the set of obstacle key point coordinate sequences, performing the following screening process steps:

carrying out back projection on each barrier key point coordinate in the barrier key point coordinate sequence to obtain a three-dimensional barrier key point coordinate sequence;

constructing a unit vector sequence by utilizing each three-dimensional barrier key point coordinate in the three-dimensional barrier key point coordinate sequence;

in response to determining that the coordinates of each barrier key point in the barrier key point coordinate sequence meet a preset key point condition and the unit vector sequence meets a preset vector relationship condition, determining the three-dimensional barrier key point coordinate sequence as a target barrier key point coordinate sequence;

wherein the right angle constraint equation is constructed by the following steps:

wherein e is ¹ The result, i.e. the right angle relation error,

representing a rectangular error, l, corresponding to a jth unit vector in the sequence of unit vectors _j Representing the jth unit vector, l, in the sequence of unit vectors _j+1 Represents the j +1 th unit vector in the unit vector sequence;

wherein the initial road surface equation is generated by:

extracting lane line points of a preset target road image to obtain a lane line point coordinate set;

back projecting each lane line point coordinate in the lane line point coordinate set to a vehicle coordinate system to obtain a lane line point three-dimensional coordinate set;

generating the initial road surface equation based on the three-dimensional coordinate set of the lane line points; before updating the initial road surface equation based on the right-angle constraint equation to obtain the target road surface equation, the method further comprises:

generating a set of key point projection error sequences based on a preset camera internal parameter matrix and a coordinate conversion matrix;

updating the initial road surface equation based on the right-angle constraint equation to obtain a target road surface equation, wherein the method comprises the following steps:

updating an initial road surface equation based on the set of the key point projection error sequence groups, the right-angle constraint equation and a preset covariance matrix to obtain a target road surface equation;

updating an initial road surface equation based on the set of the key point projection error sequence groups, the right-angle constraint equation and a preset covariance matrix to obtain a target road surface equation, wherein the method comprises the following steps:

determining an initial state vector of an initial state equation;

obtaining a target state vector sequence by the following formula:

wherein Z represents the initial state vector, Z 'represents a target state vector in a target state vector sequence, Z' _i Representing the ith target state vector, P (m), in the sequence of target state vectors _i，p，n ) Expressing the result of inputting the nth target obstacle key point coordinate in the pth target obstacle key point coordinate sequence in the ith target obstacle key point coordinate sequence group in the target obstacle key point coordinate sequence group set into the initial state equation,

representing the result of a constraint equation corresponding to the jth target obstacle key point coordinate in the jth target obstacle key point coordinate sequence group in the ith target obstacle key point coordinate sequence group in the target obstacle key point coordinate sequence group, namely the orthogonal relation error, is/are>

A transpose matrix representing the nth keypoint projection error in the kth keypoint projection error sequence in the ith keypoint projection error sequence set in the set of keypoint projection error sequences, device for selecting or keeping>

The method comprises the steps of representing an inverse matrix of a preset projection error covariance matrix, representing a projection parameter in a preset projection parameter group, representing an ith projection parameter in the preset projection parameter group, and representing a conversion parameter for shortening the length of a formula;

and determining the last target state vector in the target state vector sequence as a parameter of the target road surface equation to complete the updating of the initial road surface equation to obtain the target road surface equation.

4. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-2.

5. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-2.

Images