CN116681721B - Linear track detection and tracking method based on vision - Google Patents

Abstract

The invention provides a linear track detection and tracking method based on vision, which comprises the following steps: firstly, shooting a linear track image by using a visual image acquisition device; carrying out image enhancement on the linear track, extracting gradient characteristic information, and identifying the edge of each linear track according to the gradient amplitude value to obtain an edge map; eliminating perspective effect generated in the image shooting process by adopting an inverse perspective transformation method on the edge map to obtain a transformed edge map; detecting the edges of the transformed linear tracks by adopting a sliding window method, screening out the edges of the linear tracks meeting the requirements, and finally fitting to obtain an equation of the straight line where the edges of the linear tracks are located; and tracking each linear track meeting the requirements detected in the continuously shot images by adopting a Kalman filter, and adaptively adjusting the Kalman gain. The invention can detect and track single or multiple parallel thin or thick continuous or discontinuous straight tracks under complex working conditions.

Description

A vision-based straight line trajectory detection and tracking method

Technical Field

The invention belongs to the field of computer vision and image processing, and in particular relates to a vision-based straight line trajectory detection and tracking method.

Background Art

In applications such as vehicle assisted driving, autonomous driving, and robot line patrol, the detection and tracking of straight line trajectories is one of the key core technologies. Since most of the straight line trajectories are currently sprayed or printed on the ground with colored paint, the image of the straight line trajectory can only be acquired using visual image acquisition equipment, and then detected and tracked by processing and analyzing the image. The existing color-based straight line trajectory detection method is easily affected by lighting and weather conditions, and has poor robustness in detection and tracking in complex environments. The method based on straight line detection cannot handle situations such as straight line trajectory interruption, pollution, and wear, resulting in false detection and missed detection.

With the development of deep learning technology, straight line trajectory detection methods based on deep learning continue to emerge. However, such methods generally rely on large-scale image datasets with real labels for training. However, for many special application scenarios, such as ports and factories, the cost of collecting and producing straight line trajectory detection image datasets containing a variety of weather and lighting conditions is very high, which greatly limits the application of deep learning-based methods. In addition, the generalization ability of such methods is limited, and the model trained on a specific scene dataset cannot be applied to other scenarios.

In the context of industrial automation and intelligence, many application scenarios have put forward higher requirements for the accuracy of linear trajectory detection, which can even reach the level of tens of microns (such as weld tracking). This requires the linear trajectory detection algorithm to have stronger feature extraction and detection capabilities, and to be able to use environmental information to filter out erroneous detection results. In addition, due to the diverse application scenarios, the detection algorithm must be able to make adaptive adjustments based on factors such as lighting, weather, and the status of the marking line to improve the robustness of detection and tracking.

In summary, the existing straight line trajectory detection methods based on image processing have poor accuracy and robustness, and are difficult to meet the use requirements of complex scenes; while the straight line trajectory detection methods based on deep learning rely on large-scale data sets and have poor generalization capabilities, which limits their application in actual scenes. In response to these problems, the present invention proposes a straight line trajectory detection and tracking method based on vision, which can realize the detection and tracking of single, double, and multiple, continuous or discontinuous straight line trajectories under complex weather and working conditions.

Summary of the invention

To solve the above problems, the present invention proposes a vision-based straight line trajectory detection and tracking method, which includes the following steps:

Step S1: the visual image acquisition device captures a straight line trajectory image;

Step S2: enhancing the straight line track image captured in step S1, extracting gradient feature information, and identifying the edge of each straight line track according to the gradient amplitude information to obtain an edge map;

Step S3: using an inverse perspective transformation method to eliminate the perspective effect produced during the image shooting process on the edge map obtained in step S2, and obtaining a transformed edge map;

Step S4: using a sliding window method to detect the edges of each of the transformed straight line tracks obtained in step S3, and screening out the straight line track edges that meet the requirements, and finally fitting to obtain the equation of the straight line where the straight line track edges that meet the requirements are located;

Step S5: using a Kalman filter to track each straight line trajectory that meets the requirements detected in the multiple images taken continuously, and adaptively adjusting the Kalman gain according to the detection confidence of the edge of the straight line trajectory.

Furthermore, step S1 specifically includes the following steps:

Step 1-1: The visual image acquisition device includes but is not limited to a surveillance camera, an industrial camera and a network camera;

Step 1-2: Deploy the visual image acquisition device at an appropriate position of the vehicle or mobile robot to ensure that the straight line trajectory to be detected and tracked is within the field of view of the visual image acquisition device. The optical axis of the visual image acquisition device is perpendicular to the ground or inclined to the ground at an angle γ, γ∈[0°,90°]. Use the visual image acquisition device to capture an image of the straight line trajectory.

Furthermore, step S2 specifically includes the following steps:

Step 2-1: taking the straight line trajectory images captured by the visual image acquisition device as input one by one, and using a fixed-size rectangular frame to divide the region of interest where the straight line trajectory is located in the input image;

Step 2-2: grayscale the image in the region of interest;

Step 2-3: Perform image enhancement on the grayscale processed image;

Step 2-4: Use the Sobel operator to extract the horizontal and vertical gradient information G _x and G _y of the enhanced image respectively:

Where * represents the convolution operation, and I represents the image of the region of interest after grayscale processing and image enhancement;

Step 2-5: Calculate the gradient magnitude GM of each pixel according to the following formula:

GM(x,y)＝|G _x (x,y)|

Where G _x (x, y) represents the gradient information G _x of the pixel at the coordinate (x, y);

Step 2-6: Set the gradient amplitude threshold _TM . The pixels in the region of interest image whose gradient amplitude GM is greater than or equal to the threshold _TM are edge pixels, and the edge map E is obtained:

Furthermore, step S3 specifically includes the following steps:

Step 3-1: The process of inverse perspective transformation of the edge image obtained in step S2 is as follows:

Where A _3×3 represents the inverse perspective transformation matrix, [uv 1] ^T represents the coordinates of the pixels in the original image, which are called the source point coordinates, and [xy 1] ^T represents the coordinates of the pixels after transformation, which are called the target point coordinates. At least three sets of corresponding source and target point coordinates that are not collinear need to be obtained to solve the inverse perspective transformation matrix A _3×3 ;

Step 3-2: During initialization, for a single thin straight line track, manually select the intersection points of the edge of the straight line track and the upper and lower boundaries of the region of interest in the image, and calculate two points symmetrical about the horizontal center line of the edge map E as four source points.

For a single thick straight line trajectory, the intersection points of the left and right edges of the straight line trajectory and the upper and lower boundaries of the region of interest were selected in the image by manual selection as four source points;

For multiple parallel thin straight line tracks, the intersection points of the edge of the first straight line track from the left and the edge of the last straight line track with the upper and lower boundaries of the region of interest were selected manually in the image as four source points;

For multiple parallel thick straight line tracks, the intersection points of the left edge of the first straight line track from the left and the right edge of the last straight line track from the left and the upper and lower boundaries of the region of interest are selected manually in the image as four source points;

Starting from the source point in the lower left corner, they are named LB, LU, RU, and RB in counterclockwise order, and their coordinates are [u _LB v _LB 1] ^T , [u _LU v _LU 1] ^T , [u _RU v _RU 1] ^T , and [u _RB v _RB 1] ^T ;

Step 3-3: The coordinates of the target points corresponding to the four source points are

Step 3-4: According to the coordinates of the source point and the target point obtained in steps 3-2 and 3-3, the inverse perspective transformation matrix A _3×3 described in step 3-1 is obtained, and the homogeneous coordinates of all the pixel points in the edge map obtained in step S2 are multiplied by A _3×3 to obtain the transformed edge map;

Step 3-5: For an image containing multiple parallel straight line tracks, the source point coordinates of the current frame are automatically selected according to the straight line track detection result of the previous frame image, thereby realizing adaptive adjustment of the inverse perspective transformation matrix; affected by the perspective effect, the straight line tracks that originally maintained a parallel relationship intersect at the vanishing point in the image; using the straight line equation where the edge of the straight line track detected in the previous frame image is located, the intersection point, i.e., the vanishing point coordinates, is calculated by the least squares method; then, for the thin straight line track, the vanishing point is connected to the lower end points of the edge of the first straight line track from the left and the edge of the last straight line track, respectively; for the thick straight line track, the vanishing point is connected to the lower end points of the left edge of the first straight line track from the left and the right edge of the last straight line track, respectively; the intersection points of the two connecting lines with the upper and lower boundaries of the region of interest are a set of source points of the current frame image.

Furthermore, step S4 specifically includes the following steps:

Step 4-1: Project the edge map after inverse perspective transformation onto the horizontal axis of the image to obtain the corresponding statistical histogram;

Step 4-2: Perform mean filtering on the statistical histogram by one-dimensional convolution to eliminate the noise in the data, and only retain the points whose statistical values exceed _TE % of the maximum value of the statistical result;

Step 4-3: After filtering, the horizontal coordinate corresponding to the peak point in each histogram represents the initial base point coordinate of the current edge;

Step 4-4: Starting from the initial base point, use a fixed window size h×w and step length s to search along the vertical axis of the image, where h is the height of the window and w is the width of the window, which is numerically equal to the width threshold dividing the thick straight line and the thin straight line. If the number of edge pixels in the window exceeds the set threshold _TP , the window is considered a valid window, and the edge pixels inside it are regarded as valid edge pixels. The average value of the horizontal coordinates of all edge pixels inside the window is calculated to update the base point coordinates of the search window.

Step 4-5: If the number of edge pixels in the window does not exceed the set threshold value _TP , the window is deemed to be an invalid window, the base point coordinates are kept unchanged, and the search continues along the vertical axis until the preset search times N are reached;

Step 4-6: For each straight track edge, if the number of valid windows corresponding to it is greater than the threshold T _W , the straight track edge is identified as a candidate edge. For a single thin or thick straight track, the candidate edge is a straight track edge that meets the requirements;

For multiple parallel thin or thick straight line tracks, it is necessary to pair the straight line track edges and select the candidate edges that meet the requirements in combination with the distance constraints between the same pair of candidate edges. The specific steps include steps 4-7:

Step 4-7: For multiple parallel thin straight line trajectories, the set of initial base point coordinates of all candidate edges in the t-th edge graph Gr _t = <E _t ,E _t+1 > is respectively B _t ,B _t+1 , t=1,2,...,T-1, then the initial base point coordinates of the candidate edges must satisfy the following conditions:

Where _Dt is the fixed distance between the edges of the t-th pair of straight line tracks, and _Tt is the distance deviation threshold between the edges of the t-th pair of straight line tracks. For the initial base point coordinate set _B′t , B′t ₊₁ , t＝1, 2, ..., T-1 that meets the above conditions, the effective initial base point coordinates are

The corresponding candidate edge is a straight line trajectory edge that meets the requirements;

For multiple parallel thick straight line trajectories, the tth pair of edge graphs and The sets of initial base point coordinates of all candidate edges in are Then the initial base point coordinates of the candidate edge must meet the following conditions:

in and are the fixed distances between the left and right edges of the t-th pair of straight line tracks, T _t ^L and T _t ^R are the distance deviation thresholds between the left and right edges of the t-th pair of straight line tracks; for the initial base point coordinate set that meets the above conditions The valid initial base point coordinates are

The corresponding candidate edge is a straight line trajectory edge that meets the requirements;

Step 4-8: For each straight line trajectory edge that meets the requirements, perform straight line fitting on the valid edge pixels contained therein to obtain the straight line equation corresponding to the edge.

Furthermore, the specific method for pairing the edges of the straight line tracks is:

For multiple parallel thin straight line tracks, the straight line tracks are divided into T straight line tracks from left to right along the horizontal direction of the image, where T represents the number of straight line tracks, and the edge map E is saved as T edge maps E ₁ , E ₂ , ..., E _T according to the straight line tracks to which the edges belong:

According to the relative position relationship, the edges of two adjacent straight track are divided into a pair of Gr _t , and the division method is as follows:

Gr _t =<E _t ,E _t+1 >,t＝1,2,...,T-1

For the rough straight line track in the input image, the straight line track edge is divided into three categories: left side, right side and others according to the gradient direction information. The specific steps are as follows:

The gradient direction feature GD of each pixel is calculated according to the following formula:

Where G _x (x, y) and G _y (x, y) represent the gradient information G _x and G _y of the pixel at the coordinate (x, y) respectively;

According to the directional feature GD, all the pixels in the edge map E are divided into the left and right Right and others The three sets are divided as follows:

Other The edge pixels of the category will be ignored, that is, if Then E(x,y)＝0; belongs to the left side The pixel points of the category are saved as the left edge map ^EL , that is, if Then E ^L (x, y) = 255; belongs to the right side The pixel points of the category are saved as the right edge map ^ER , that is, if Then ^ER (x, y) = 255;

For multiple parallel thick straight line tracks, divide the straight line tracks into T straight line tracks from left to right along the horizontal direction of the image, where T represents the number of straight line tracks, and save the edge map E as 2T edge maps according to the straight line track and edge category to which the edge belongs.

According to the relative position relationship, the edges of the same category of two adjacent straight line trajectories are divided into a pair or The division is as follows:

Furthermore, the effective pixel number threshold _TP in steps 4-4 and 4-5 is adaptively adjusted according to the straight line trajectory detection result and the number of searches:

Where N represents the preset number of searches, P _x represents the sum of the statistical values of valid edge pixels within the search window, and

in, It represents the statistical value corresponding to the coordinate i in the statistical histogram, and x represents the horizontal coordinate value of the initial base point.

Furthermore, step S5 specifically includes the following steps:

Step 5-1: The number of valid windows described in step 4-6 reflects the reliability of the current candidate edge detection. The more valid windows there are, the higher the confidence of the candidate edge detection.

For thin straight line trajectories, the detection confidence is calculated as:

Where c _t represents the detection confidence corresponding to the tth straight line track edge that meets the requirements, _nt represents the number of valid windows of the edge, N represents the preset number of searches, β represents the correction coefficient, and T represents the total number of straight line tracks that meet the requirements;

For a rough straight line trajectory, the detection confidence is calculated as:

in and They represent the detection confidence corresponding to the left and right edges of the tth straight line trajectory that meets the requirements, and They represent the number of valid windows on the left and right edges of the tth straight line trajectory that meets the requirements, N represents the preset number of searches, β represents the correction coefficient, and T represents the total number of straight line trajectories that meet the requirements;

Step 5-2: For thin straight line trajectories, the straight line where the edge of each straight line trajectory that meets the requirements is located and the upper and lower boundaries of the region of interest form two intersection points, and their corresponding confidence levels are and and Then construct the correction coefficient matrix according to the confidence level

For the rough straight line trajectory, the straight lines where the left and right edges of each straight line trajectory that meets the requirements are located and the upper and lower boundaries of the region of interest form four intersection points, and their corresponding confidence levels are and and Then construct the correction coefficient matrix according to the confidence level

Step 5-3: Calculate the corrected measurement error covariance matrix R'

R' = diag(C _k )R

Where diag(·) represents the diagonalization operation, R represents the initial measurement error covariance matrix, R = s _R × U, s _R represents the scaling factor, and U represents the unit matrix;

Step 5-4: The modified Kalman gain K _k is

K _k =P _k M ^T (MP _k M ^T +R') ^-1

Where _Pk represents the covariance matrix of the prior estimation error of the k-th frame image, initialized to _P0 = U, M represents the measurement matrix, for a thin straight line trajectory, For a rough straight line trajectory, The element _Mij in the i-th row and j-th column is as follows

Step 5-5: The calculation method of the covariance matrix P _k of the prior estimation error in step 5-4 is:

Where Q represents the covariance matrix of process noise, Q = s _Q × U, s _Q represents the scaling factor, the superscript T represents the transpose of the matrix, and A represents the transfer state matrix.

For thin straight lines:

For a rough straight line trajectory:

Represents the covariance matrix of the posterior estimation error of the k-1th frame, and its update process is as follows

Where U represents the identity matrix;

Step 5-6: For the thin straight line trajectory, the straight line where the edge of the tth straight line trajectory that meets the requirements obtained in step 4-8 is located forms two intersection points with the upper and lower boundaries of the region of interest. The horizontal coordinates of the intersection points are and Construct the k-th frame measurement matrix

For the rough straight line trajectory, the straight lines where the left and right edges of the tth straight line trajectory that meets the requirements obtained in steps 4-8 and the upper and lower boundaries of the region of interest form four intersection points, and the horizontal coordinates of the intersection points are and Construct the k-th frame measurement matrix

The k-th frame measurement matrix Z _k is calculated as follows:

Z _k =MX _k

Where _Xk represents the state estimation matrix of the kth frame;

Steps 5-7: For thin straight track: in and They respectively represent the estimated values of the horizontal coordinates of the two intersection points formed by the straight line where the edge of the t-th straight line trajectory is located and the upper and lower boundaries of the region of interest, which are estimated from the prediction result of the previous frame image; Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between

For thick straight line tracks, the width of the straight line track will be used as an additional constraint to track the straight line track: in and The estimated values of the horizontal coordinates of the four intersection points formed by the straight lines where the left and right edges of the t-th straight line trajectory are located and the upper and lower boundaries of the region of interest are estimated from the prediction results of the previous frame image; Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between represents the width of the top of the straight line track, Indicates the change in width between two adjacent frames. Indicates the width of the bottom of the straight line track, Indicates the change in width between two adjacent frames;

Step 5-8: The k-th frame state estimation matrix X _k is calculated as follows:

in Represents the posterior state estimate matrix of the k-1th frame, which is calculated as follows:

For thin straight line trajectories, the initial value is the detection result of the horizontal coordinate of the edge endpoint of the straight line trajectory of the first frame image; for the rough straight line trajectory, the initial value is the detection result of the horizontal coordinates of the edge endpoints of the straight line track and the width of the straight line track of the first frame image;

Step 5-9: The final output of the straight line trajectory tracking is The calculation is as follows:

Beneficial effects: The gradient feature extraction method based on image enhancement can enhance the robustness of the straight line trajectory edge detection to changes in lighting and weather, and improve the adaptability to complex environments; the straight line trajectory detection method based on sliding windows can effectively solve problems such as marker line interruption, pollution, and wear; according to the straight line trajectory detection results, the Kalman gain is adaptively adjusted to enhance the stability of lane line tracking. The straight line trajectory detection method proposed by the present invention does not need to rely on data set training, and has high detection accuracy and robustness, and can realize the detection and tracking of single or multiple parallel thin or thick continuous or discontinuous straight line trajectories in complex environments and working conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of the installation of a monitoring camera on a port rubber-tyred gantry crane;

Figure 2 is an image of a straight line track (lane line) captured by a surveillance camera;

FIG3 is a flow chart of the present invention;

Fig. 4 is an edge detection result diagram;

FIG5 is a right edge diagram of the second straight line track after inverse perspective transformation;

FIG6 is a schematic diagram of a method for selecting source point coordinates based on vanishing points;

FIG7 is a statistical histogram corresponding to the right edge of the second straight line track after filtering;

FIG8 is a statistical histogram corresponding to the right edge of the second straight line track;

FIG9 is a schematic diagram of a method for detecting a straight line trajectory based on a sliding window method;

FIG10 is a schematic diagram of the structure of a mobile robot;

Figure 11 is a weld image taken by an industrial camera.

DETAILED DESCRIPTION

In order to fully explain the detection and tracking process of the present invention under different visual acquisition equipment deployment modes and different straight line trajectory conditions, the present invention introduces two specific embodiments: port tire gantry crane lane line detection and mobile robot weld tracking. The present invention is further described in detail below with reference to the accompanying drawings.

Invention Example 1:

As shown in FIG1 , the vision-based straight-line trajectory detection and tracking method of the present invention is based on the hardware foundation in the embodiment of lane line detection of a port tire-type gantry crane as follows: the column 2 of the port tire-type gantry crane is located in the middle of two straight-line trajectories 3, and the visual image acquisition device 1 is installed on the column 2 of the port tire-type gantry crane, and the optical axis of the device 1 forms an angle of 45° with the column and is inclined toward the ground.

Based on the above hardware system, the visual-based straight line trajectory detection and tracking method of this embodiment includes the following steps:

Step 1: The visual image acquisition device captures the straight line trajectory image;

Step 1-1: The visual image acquisition device includes but is not limited to a surveillance camera, an industrial camera, and a network camera. In this embodiment, a surveillance camera is used;

Step 1-2: Deploy the surveillance camera on the column of the port rubber-tyred gantry crane to ensure that the straight line trajectory to be detected and tracked is within the field of view of the surveillance camera. The optical axis of the surveillance camera forms an angle of γ = 45° with the column and is tilted toward the ground.

Step 1-3: Use a surveillance camera to capture an image of a straight line trajectory, as shown in Figure 2. In the embodiment of the present invention, the straight line trajectory to be detected and tracked is two lane lines, which are two parallel yellow straight lines with a width of 12 cm and a distance of 60 cm between the inner edges of the lane lines; the measured width of each lane line in the horizontal direction of the image is 25 pixels, which is greater than the threshold value w=15 pixels set in the embodiment of the present invention, so the straight line trajectory belongs to a plurality of parallel thick straight line trajectories.

The flow chart of the method involved in the present invention is shown in FIG3 , and comprises the following steps:

Step 2: Enhance the straight line trajectory image captured in step 1, extract gradient feature information, and identify the edge of each straight line trajectory according to the gradient amplitude information to obtain an edge map;

Step 2-1: taking the straight line trajectory images captured by the visual image acquisition device as input one by one, and using a fixed-size rectangular frame to divide the region of interest where the straight line trajectory is located in the input image;

Step 2-2: grayscale the image in the region of interest. The grayscale processing method used in this embodiment is as follows:

I(x,y)＝λ ₁ R(x,y)+λ ₂ G(x,y)+λ ₃ B(x,y)

Wherein the weight vector λ = [λ ₁ ,λ ₂ ,λ ₃ ] = [0.3,0.3,-0.3], R(x,y), G(x,y), B(x,y) represent the R, G, B color components corresponding to the pixel with coordinates (x,y), and I(x,y) represents the gray value after transformation;

Step 2-3: Perform image enhancement on the grayscaled image. This embodiment uses a contrast-limited adaptive histogram equalization algorithm (CLAHE) to perform image enhancement processing. By dividing the entire image into several regions, performing histogram equalization on each region separately, and then fusing them using bilinear interpolation, the contrast of the image can be effectively improved.

Step 2-4: Use the Sobel operator to extract the gradient information G _x and G _y in the horizontal and vertical directions of the enhanced image respectively:

Where * represents the convolution operation, and I represents the image of the region of interest after grayscale processing and image enhancement;

Step 2-5: Calculate the gradient magnitude GM of each pixel according to the following formula:

GM(x,y)＝|G _x (x,y)|

Where G _x (x, y) represents the gradient information G _x of the pixel at the coordinate (x, y);

Step 2-6: Set the gradient amplitude threshold _TM . In the embodiment of the present invention, _TM = 20. The pixels in the image of the region of interest whose gradient amplitude GM is greater than or equal to the threshold _TM are edge pixels. The edge map E is obtained, as shown in FIG4 :

For the rough straight line track, the straight line track edge is divided into three categories: left side, right side and others according to the gradient direction information. The specific steps are as follows:

The gradient direction feature GD of each pixel is calculated according to the following formula:

Where G _x (x, y) and G _y (x, y) represent the gradient information G _x and G _y of the pixel at the coordinate (x, y) respectively;

According to the directional feature GD, all the pixels in the edge map E are divided into the left and right Right and others The three sets are divided as follows:

In the embodiment of the present invention, Other The edge pixels of the category will be ignored, that is, if Then E(x,y)＝0; belongs to the left side The pixel points of the category are saved as the left edge map ^EL , that is, if Then E ^L (x, y) = 255; belongs to the right side The pixel points of the category are saved as the right edge map ^ER , that is, if Then ^ER (x, y) = 255;

For multiple parallel thick straight line tracks, the straight line tracks are divided into T straight line tracks from left to right along the horizontal direction of the image, where T represents the number of straight line tracks. In the embodiment of the present invention, T=2, and the edge map E is saved as 4 edge maps according to the straight line track to which the edge belongs and the edge category.

According to the relative position relationship, the edges of the same category of two adjacent straight line trajectories are divided into a pair or The division is as follows:

Step 3: Using the inverse perspective transformation method on the edge map obtained in step 2 to eliminate the perspective effect generated during the image shooting process, and obtain a transformed edge map;

Step 3-1: The process of inverse perspective transformation of the edge image obtained in step 2 is as follows:

Where A _3×3 represents the inverse perspective transformation matrix, [uv 1] ^T represents the coordinates of the pixels in the original image, which are called the source point coordinates, and [xy 1] ^T represents the coordinates of the pixels after transformation, which are called the target point coordinates. At least three sets of corresponding source and target point coordinates that are not collinear need to be obtained to solve the inverse perspective transformation matrix A _3×3 ;

Step 3-2: During initialization, for multiple parallel thick straight line tracks, manually select the intersection points of the left edge of the first straight line track from the left and the right edge of the last straight line track with the upper and lower boundaries of the region of interest in the image as four source points;

Starting from the source point in the lower left corner, they are named LB, LU, RU, and RB in counterclockwise order, and their coordinates are [u _LB v _LB 1] ^T , [u _LU v _LU 1] ^T , [u _RU v _RU 1] ^T , and [u _RB v _RB 1] ^T ;

Step 3-3: The coordinates of the target points corresponding to the four source points are

Step 3-4: According to the coordinates of the source point and the target point obtained in steps 3-2 and 3-3, the inverse perspective transformation matrix A _3×3 described in step 3-1 is obtained, and the homogeneous coordinates of all the pixel points in the edge map obtained in step S2 are multiplied by A _3×3 to obtain the transformed edge map; taking the right edge of the second straight line trajectory as an example, the transformed edge map is shown in FIG5 , in which the horizontal and vertical axes represent the horizontal and vertical coordinates of the image respectively;

Step 3-5: For an image containing multiple parallel straight line tracks, the source point coordinates of the current frame are automatically selected according to the straight line track detection result of the previous frame image, thereby realizing adaptive adjustment of the inverse perspective transformation matrix; as shown in FIG6 , affected by the perspective effect, the straight line tracks that originally maintained a parallel relationship intersect at the vanishing point in the image (the five-pointed star mark in the figure); using the straight line equation where the edge of the straight line track detected in the previous frame image is located, the intersection point, i.e., the vanishing point coordinates, is calculated by the least squares method; then for the rough straight line track, the vanishing point is connected to the left edge of the first straight line track from the left and the lower end point of the right edge of the last straight line track respectively; the intersection point (the cross mark in the figure) of the two connecting lines and the upper and lower boundaries of the region of interest (the rectangular box in the figure) is a set of source points of the current frame image.

Step 4: Use the sliding window method to detect the edges of the transformed straight line tracks obtained in step S3 one by one, and select the straight line track edges that meet the requirements, and finally fit the equation of the straight line where the straight line track edges that meet the requirements are located;

Step 4-1: Transform the edge map after inverse perspective Project the images onto the horizontal axis to obtain the corresponding statistical histograms. Taking the right edge of the second straight line track as an example, the statistical histogram is shown in FIG7 , where the horizontal axis represents the horizontal axis of the image, and the vertical axis represents the statistical value of the number of edge pixels, in units of pieces.

Step 4-2: Perform mean filtering on the statistical histogram by means of one-dimensional convolution to eliminate the noise in the data, and only retain the points whose statistical values exceed _TE % of the maximum value of the statistical result. In the embodiment of the present invention, _TE %=25%. Taking the right edge of the second straight line track as an example, the filtered histogram is shown in FIG8 , where the horizontal axis represents the horizontal axis of the image, and the vertical axis represents the statistical value of the number of edge pixel points, in units of pieces;

Step 4-3: After filtering, the horizontal coordinate corresponding to the peak point in each histogram represents the initial base point coordinate of the current edge;

Step 4-4: Starting from the initial base point, search along the vertical axis of the image using a fixed window size h×w and step length s, where h is the height of the window and w is the width of the window, which is numerically equal to the width threshold dividing the thick straight line and the thin straight line. In the embodiment of the present invention, w=15 (pixels), the height of the window h is equal to the step length s, and the calculation method is as follows

Wherein H represents the height of the region of interest, N represents the preset number of searches, and in the embodiment of the present invention, H=344 (pixels), N=10;

If the number of edge pixels in the window exceeds the set threshold _TP , the window is considered a valid window, and the edge pixels inside it are regarded as valid edge pixels. The average value of the horizontal coordinates of all edge pixels inside the window is calculated to update the base point coordinates of the search window. The threshold _TP for the number of valid pixels can be adaptively adjusted according to the straight line trajectory detection results and the number of searches:

in

in, represents the statistical value corresponding to the coordinate i in the statistical histogram, and x represents the horizontal coordinate value of the initial base point;

Step 4-5: If the number of edge pixels in the window does not exceed the set threshold value _TP , the window is deemed to be an invalid window, the base point coordinates are kept unchanged, and the search continues along the vertical axis until the preset search times N are reached, as shown in FIG9 , where the horizontal and vertical axes represent the horizontal and vertical coordinates of the image respectively;

Step 4-6: For each straight track edge, if the number of valid windows corresponding to it is greater than the threshold value T _W , in the embodiment of the present invention, T _W = 4, then the straight track edge is identified as a candidate edge; for multiple parallel thick straight track edges, it is necessary to pair the straight track edges, and combine the distance constraints between the same pair of candidate edges to screen out the candidate edges that meet the requirements, which specifically includes step 4-7:

Step 4-7: The candidate edges described in step 4-6 may have the result of false detection. If the distances between multiple straight line trajectories are fixed, the falsely detected candidate edges can be screened out by using the distance constraints between the candidate edges. The sets of initial base point coordinates of all candidate edges in the four edge images are respectively Then the initial base point coordinates of the candidate edge must meet the following conditions:

in and are respectively the fixed distances between the left and right edges of the first pair of straight track, T ₁ ^L and T ₁ ^R are respectively the distance deviation thresholds between the left and right edges of the first pair of straight track; For the initial base point coordinate set that meets the above conditions The valid initial base point coordinates are

The corresponding candidate edge is a straight line trajectory edge that meets the requirements;

Step 4-8: For each straight line trajectory edge that meets the requirements, perform straight line fitting on the valid edge pixels contained therein to obtain the straight line equation corresponding to the edge.

Step 5: Use the Kalman filter to track each straight line trajectory that meets the requirements detected in multiple images taken continuously, and adaptively adjust the Kalman gain according to the detection confidence of the edge of the straight line trajectory.

Step 5-1: The number of valid windows described in step 4-6 reflects the reliability of the current candidate edge detection. The more valid windows there are, the higher the confidence of the candidate edge detection. For a rough straight line trajectory, the calculation method of the detection confidence is:

in and They represent the detection confidence corresponding to the left and right edges of the tth straight line trajectory that meets the requirements, and represents the number of valid windows on the left and right edges of the t-th straight line track that meets the requirements, N represents the preset number of searches, β represents the correction coefficient, and β=100 is taken in the embodiment of the present invention;

Step 5-2: For the rough straight line trajectory, the left and right edges of each straight line trajectory that meets the requirements form four intersection points with the upper and lower boundaries of the region of interest, and their corresponding confidence levels are and and Then construct the correction coefficient matrix according to the confidence level

Step 5-3: Calculate the corrected measurement error covariance matrix R'

R' = diag(C _k )R

Wherein diag(·) represents a diagonalization operation, R represents the initial measurement error covariance matrix, R=s _R ×U, s _R represents a scaling factor, and in the embodiment of the present invention, s _R =500 is taken, and U represents a unit matrix;

Step 5-4: The corrected Kalman gain is

K _k =P _k M ^T (MP _k M ^T +R') ^-1

Where P _k represents the covariance matrix of the prior estimation error of the k-th frame image, which is initialized to P ₀ = U, Represents the measurement matrix, where the element _Mij in the i-th row and j-th column is as follows

Step 5-5: The calculation method of the covariance matrix P _k of the prior estimation error in step 5-4 is:

Where Q represents the covariance matrix of process noise, Q = s _Q × U, s _Q represents the scaling factor, and in the embodiment of the present invention, s _Q = 0.1. A represents the transfer state matrix

Represents the covariance matrix of the posterior estimation error of the k-1th frame, and its update process is as follows

Where U represents the identity matrix;

Step 5-6: For the rough straight line trajectory, the straight lines where the left and right edges of the tth straight line trajectory that meets the requirements obtained in step 4-8 and the upper and lower boundaries of the region of interest form four intersection points, and the horizontal coordinates of the intersection points are and Construct the k-th frame measurement matrix

The calculation method of the K-th frame measurement matrix Z _k is as follows:

Z _k =MX _k ;

Step 5-7: For the thick straight line track, the width value of the straight line track will be used as an additional constraint to track the straight line track: in and The estimated values of the horizontal coordinates of the four intersection points formed by the straight lines where the left and right edges of the t-th straight line trajectory are located and the upper and lower boundaries of the region of interest are estimated from the prediction results of the previous frame image; Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates the width of the top of the straight line track, Indicates the change in width between two adjacent frames. Indicates the width of the bottom of the straight line track, Indicates the change in width between two adjacent frames;

Step 5-8: The k-th frame state estimation matrix X _k is calculated as follows:

Represents the posterior state estimate matrix of the K-1th frame, which is calculated as follows:

Initial Value is the detection result of the horizontal coordinates of the edge endpoints of the straight line track and the width of the straight line track of the first frame image;

Step 5-9: The final output of the straight line trajectory tracking is The calculation is as follows:

Invention Example 2:

As shown in Figure 10, the vision-based straight line trajectory detection and tracking method of the present invention is based on the hardware foundation: the visual image acquisition device 1 is installed on the crossbeam of the mobile robot platform 2, the optical axis of the visual image acquisition device 1 is perpendicular to the ground, and the straight line trajectory 3 is located in the center of the field of view of the visual image acquisition device 1.

Based on the above hardware system, the visual-based straight line trajectory detection and tracking method of this embodiment includes the following steps:

Step 1: The visual image acquisition device captures the straight line trajectory image;

Step 1-1: The visual image acquisition device includes but is not limited to a surveillance camera, an industrial camera and a network camera. In the embodiment of the present invention, an industrial camera is used;

Step 1-2: Deploy an industrial camera on the crossbar of the mobile robot to ensure that the linear trajectory to be detected and tracked is within the field of view of the industrial camera, and the optical axis of the industrial camera is perpendicular to the ground; use the industrial camera to capture an image of the linear trajectory, as shown in Figure 11. In the embodiment of the present invention, the linear trajectory to be detected and tracked is a single weld seam with a weld seam width of 0.1-0.2 mm. The measured width of the weld seam in the horizontal direction of the image is 10 pixels, which is less than the threshold value w=60 (pixels), so the linear trajectory belongs to a single thin linear trajectory.

The flow chart of the method involved in the present invention is shown in FIG3 , and comprises the following steps:

Step 2: Enhance the straight line track image captured in step S1, extract gradient feature information, and identify the edge of each straight line track according to the gradient amplitude information to obtain an edge map;

Step 2-1: taking the straight line trajectory images captured by the visual image acquisition device as input one by one, and using a fixed-size rectangular frame to divide the region of interest where the straight line trajectory is located in the input image;

Step 2-2: grayscale the image in the region of interest. The grayscale processing method used in this embodiment is as follows:

I(x,y)＝λ ₁ R(x,y)+λ ₂ G(x,y)+λ ₃ B(x,y)

Wherein the weight vector λ = [λ ₁ ,λ ₂ ,λ ₃ ] = [0.3,0.3,-0.3], R(x,y), G(x,y), B(x,y) represent the R, G, B color components corresponding to the pixel with coordinates (x,y), and I(x,y) represents the gray value after transformation;

Step 2-3: Perform image enhancement on the grayscaled image. This embodiment uses a contrast-limited adaptive histogram equalization algorithm (CLAHE) to perform image enhancement processing. By dividing the entire image into several regions, performing histogram equalization on each region separately, and then fusing them using bilinear interpolation, the contrast of the image can be effectively improved.

The Sobel operator is used to extract the gradient information G _x and G _y in the horizontal and vertical directions of the enhanced image respectively:

Where * represents the convolution operation, and I represents the image of the region of interest after grayscale processing and image enhancement;

Step 2-5: Calculate the gradient magnitude GM of each pixel according to the following formula:

GM(x,y)＝|G _x (x,y)|

Where G _x (x, y) represents the gradient information G _x of the pixel at the coordinate (x, y);

Step 2-6: Set the gradient amplitude threshold _TM . The pixels in the region of interest image whose gradient amplitude GM is greater than or equal to the threshold _TM are edge pixels, and the edge map E is obtained:

Step 3: Since the camera optical axis is perpendicular to the ground, the plane where the weld is located is parallel to the camera imaging plane and is not affected by the perspective effect, so there is no need to perform inverse perspective transformation;

Step 4: Use the sliding window method to detect the edges of the transformed straight line tracks obtained in step S3 one by one, and select the straight line track edges that meet the requirements, and finally fit the equation of the straight line where the straight line track edges that meet the requirements are located;

Step 4-1: Project the edge map E onto the horizontal axis of the image to obtain the corresponding statistical histogram;

Step 4-2: Perform mean filtering on the statistical histogram by one-dimensional convolution to eliminate the noise in the data, and only retain the points whose statistical values exceed _TE % of the maximum value of the statistical result. In the embodiment of the present invention, _TE %=25%;

Step 4-3: After filtering, the horizontal coordinate corresponding to the peak point in each histogram represents the initial base point coordinate of the current edge;

Step 4-4: Starting from the initial base point, search along the vertical axis of the image using a fixed window size h×w and step length s, where h is the height of the window and w is the width of the window, which is numerically equal to the width threshold dividing the thick straight line and the thin straight line. In the embodiment of the present invention, the width of the search window w=60 (pixels), the height of the window h is equal to the step length s, and the calculation method is as follows

Wherein H represents the height of the region of interest, N represents the preset number of searches, and in the embodiment of the present invention, H=2592 (pixels), N=10;

If the number of edge pixels in the window exceeds the set threshold _TP , the window is considered a valid window, and the edge pixels inside it are considered valid edge pixels. The average value of the horizontal coordinates of all edge pixels inside the window is calculated to update the base point coordinates of the search window; the threshold _TP of the number of valid pixels can be adaptively adjusted according to the straight line trajectory detection results and the number of searches:

in

in, represents the statistical value corresponding to the coordinate i in the statistical histogram, and x represents the horizontal coordinate value of the initial base point;

Step 4-5: If the number of edge pixels in the window does not exceed the set threshold value _TP , the window is deemed to be an invalid window, the base point coordinates are kept unchanged, and the search continues along the vertical axis until the preset search times N are reached;

Step 4-6: For each straight track edge, if the number of valid windows corresponding to it is greater than the threshold value T _W , in the embodiment of the present invention, T _W = 4, then the straight track edge is identified as a candidate straight track edge; for a single thin straight track, the candidate edge is a straight track edge that meets the requirements;

Step 4-8: For each straight line trajectory edge that meets the requirements, perform straight line fitting on the valid edge pixels contained therein to obtain the straight line equation corresponding to the edge.

Step 5: Use the Kalman filter to track the straight line trajectory in the continuous image, and adjust the Kalman gain according to the detection confidence of the edge of the straight line trajectory;

Step 5-1: The number of valid windows described in step 4-6 reflects the reliability of the current candidate edge detection. The more valid windows there are, the higher the confidence of the candidate edge detection. For a thin straight line trajectory, the calculation method of the detection confidence is:

Wherein c ₁ represents the detection confidence corresponding to the edge of the straight track that meets the requirements, n ₁ represents the number of valid windows of the edge, N represents the preset number of searches, β represents the correction coefficient, and in the embodiment of the present invention, β=100;

Step 5-2: For the thin straight line trajectory, the straight line where the edge of the straight line trajectory meets the requirements and the upper and lower boundaries of the region of interest form two intersection points, and their corresponding confidence levels are and and Then construct the correction coefficient matrix according to the confidence level

Step 5-3: Calculate the corrected measurement error covariance matrix R'

R' = diag(C _k )R

Wherein diag(·) represents a diagonalization operation, R represents the initial measurement error covariance matrix, R=s _R ×U, s _R represents a scaling factor, and in the embodiment of the present invention, s _R =500 is taken, and U represents a unit matrix;

Step 5-4: The corrected Kalman gain is

K _k =P _k M ^T (MP _k M ^T +R') ^-1

Where _Pk represents the covariance matrix of the prior estimation error of the k-th frame image, which is initialized to _P0 = U. Represents the measurement matrix

Step 5-5: The calculation method of the covariance matrix P _k of the prior estimation error in step 5-4 is:

Where Q represents the covariance matrix of process noise, Q = s _Q × U, s _Q represents the scaling factor, and in the embodiment of the present invention, s _Q = 0.1. A represents the transfer state matrix

Represents the covariance matrix of the posterior estimation error of the k-1th frame, and its update process is as follows

Where U represents the identity matrix;

Step 5-6: For the thin straight line trajectory, the straight line where the edge of the straight line trajectory that meets the requirements obtained in step 4-8 is located forms two intersection points with the upper and lower boundaries of the area of interest. The horizontal coordinates of the intersection points are and Construct the k-th frame measurement matrix The calculation method of the K-th frame measurement matrix Z _k is as follows:

Z _k =MX _k

Where _Xk represents the state estimation matrix of the kth frame;

Steps 5-7: For thin straight track: in and The estimated values of the horizontal coordinates of the two intersection points formed by the straight line where the edge of the straight track is located and the upper and lower boundaries of the region of interest are estimated from the prediction result of the previous frame image; Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between

Step 5-8: The k-th frame state estimation matrix X _k is calculated as follows:

in Represents the posterior state estimate matrix of the k-1th frame, which is calculated as follows:

For thin straight line trajectories, the initial value is the detection result of the horizontal coordinates of the edge endpoints of the straight line track of the first frame image;

Step 5-9: The final output of the straight line trajectory tracking is The calculation is as follows:

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than to limit its protection scope. Although the present application has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present application, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims (7)

1. A method for detecting and tracking a linear trajectory based on vision, characterized in that the method is used to detect and track a single or multiple parallel thin or thick continuous or discontinuous linear trajectories, and each linear trajectory is regarded as a thick linear trajectory if its width in the horizontal direction of the image is greater than a threshold value w, otherwise, it is regarded as a thin linear trajectory; the method comprises the following steps: Step S1: the visual image acquisition device captures a straight line trajectory image; Step S2: enhancing the straight line track image captured in step S1, extracting gradient feature information, and identifying the edge of each straight line track according to the gradient amplitude information to obtain an edge map; Step S3: using an inverse perspective transformation method to eliminate the perspective effect produced during the image shooting process on the edge map obtained in step S2, and obtaining a transformed edge map; Step S4: using a sliding window method to detect the edges of each of the transformed straight line tracks obtained in step S3, and screening out the straight line track edges that meet the requirements, and finally fitting to obtain the equation of the straight line where the straight line track edges that meet the requirements are located; Step S5: using a Kalman filter to track each straight line trajectory that meets the requirements detected in the multiple images taken continuously, and adaptively adjusting the Kalman gain according to the detection confidence of the edge of the straight line trajectory; Wherein, step S3 specifically includes the following steps: Step 3-1: The process of inverse perspective transformation of the edge image obtained in step S2 is as follows: Where A _3×3 represents the inverse perspective transformation matrix, [uv 1] ^T represents the coordinates of the pixels in the original image, which are called the source point coordinates, and [xy 1] ^T represents the coordinates of the pixels after transformation, which are called the target point coordinates. At least three sets of corresponding source and target point coordinates that are not collinear need to be obtained to solve the inverse perspective transformation matrix A _3×3 ; Step 3-2: During initialization, for a single thin straight line track, manually select the intersection points of the edge of the straight line track and the upper and lower boundaries of the region of interest in the image, and calculate two points symmetrical about the horizontal center line of the edge map E as four source points. For a single thick straight line trajectory, the intersection points of the left and right edges of the straight line trajectory and the upper and lower boundaries of the region of interest were selected in the image by manual selection as four source points; For multiple parallel thin straight line tracks, the intersection points of the edge of the first straight line track from the left and the edge of the last straight line track with the upper and lower boundaries of the region of interest were selected manually in the image as four source points; For multiple parallel thick straight line tracks, the intersection points of the left edge of the first straight line track from the left and the right edge of the last straight line track from the left and the upper and lower boundaries of the region of interest are selected manually in the image as four source points; Starting from the source point in the lower left corner, they are named LB, LU, RU, and RB in counterclockwise order, and their coordinates are [u _LB v _LB 1] ^T , [u _LU v _LU 1] ^T , [u _RU v _RU 1] ^T , and [u _RB v _RB 1] ^T ; Step 3-3: The coordinates of the target points corresponding to the four source points are Step 3-4: According to the coordinates of the source point and the target point obtained in steps 3-2 and 3-3, the inverse perspective transformation matrix A _3×3 described in step 3-1 is obtained, and the homogeneous coordinates of all the pixel points in the edge map obtained in step S2 are multiplied by A _3×3 to obtain the transformed edge map; Step 3-5: For an image containing multiple parallel straight line tracks, the source point coordinates of the current frame are automatically selected according to the straight line track detection result of the previous frame image, thereby realizing adaptive adjustment of the inverse perspective transformation matrix; affected by the perspective effect, the straight line tracks that originally maintained a parallel relationship intersect at the vanishing point in the image; using the straight line equation where the edge of the straight line track detected in the previous frame image is located, the intersection point, i.e., the vanishing point coordinates, is calculated by the least squares method; then, for the thin straight line track, the vanishing point is connected to the lower end points of the edge of the first straight line track from the left and the edge of the last straight line track, respectively; for the thick straight line track, the vanishing point is connected to the lower end points of the left edge of the first straight line track from the left and the right edge of the last straight line track, respectively; the intersection points of the two connecting lines with the upper and lower boundaries of the region of interest are a set of source points of the current frame image. 2. The method for detecting and tracking a straight line trajectory based on vision according to claim 1, characterized in that step S1 specifically comprises the following steps: Step 1-1: The visual image acquisition device includes but is not limited to a surveillance camera, an industrial camera and a network camera; Step 1-2: Deploy the visual image acquisition device at an appropriate position of the vehicle or mobile robot to ensure that the straight line trajectory to be detected and tracked is within the field of view of the visual image acquisition device. The optical axis of the visual image acquisition device is perpendicular to the ground or inclined to the ground at an angle γ, γ∈[0°,90°]. Use the visual image acquisition device to capture an image of the straight line trajectory. 3. The method for detecting and tracking a straight line trajectory based on vision according to claim 1, characterized in that step S2 specifically comprises the following steps: Step 2-1: taking the straight line trajectory images captured by the visual image acquisition device as input one by one, and using a fixed-size rectangular frame to divide the region of interest where the straight line trajectory is located in the input image; Step 2-2: grayscale the image in the region of interest; Step 2-3: Perform image enhancement on the grayscale processed image; Step 2-4: Use the Sobel operator to extract the horizontal and vertical gradient information G _x and G _y of the enhanced image respectively: Where * represents the convolution operation, and I represents the image of the region of interest after grayscale processing and image enhancement; Step 2-5: Calculate the gradient magnitude GM of each pixel according to the following formula: GM(x,y)＝|G _x (x,y)| Where G _x (x, y) represents the gradient information G _x of the pixel at the coordinate (x, y); Step 2-6: Set the gradient amplitude threshold _TM . The pixels in the region of interest image whose gradient amplitude GM is greater than or equal to the threshold _TM are edge pixels, and the edge map E is obtained: 4. The method for detecting and tracking a straight line trajectory based on vision according to claim 1, wherein step S4 specifically comprises the following steps: Step 4-1: Project the edge map after inverse perspective transformation onto the horizontal axis of the image to obtain the corresponding statistical histogram; Step 4-2: Perform mean filtering on the statistical histogram by one-dimensional convolution to eliminate the noise in the data, and only retain the points whose statistical values exceed _TE % of the maximum value of the statistical result; Step 4-3: After filtering, the horizontal coordinate corresponding to the peak point in each histogram represents the initial base point coordinate of the current edge; Step 4-4: Starting from the initial base point, use a fixed window size h×w and step length s to search along the vertical axis of the image, where h is the height of the window and w is the width of the window, which is numerically equal to the width threshold dividing the thick straight line and the thin straight line. If the number of edge pixels in the window exceeds the set threshold _TP , the window is considered a valid window, and the edge pixels inside it are regarded as valid edge pixels. The average value of the horizontal coordinates of all edge pixels inside the window is calculated to update the base point coordinates of the search window. Step 4-5: If the number of edge pixels in the window does not exceed the set threshold value _TP , the window is deemed to be an invalid window, the base point coordinates are kept unchanged, and the search continues along the vertical axis until the preset search times N are reached; Step 4-6: For each straight track edge, if the number of valid windows corresponding to it is greater than the threshold T _W , the straight track edge is identified as a candidate edge. For a single thin or thick straight track, the candidate edge is a straight track edge that meets the requirements; For multiple parallel thin or thick straight line tracks, it is necessary to pair the straight line track edges and select the candidate edges that meet the requirements in combination with the distance constraints between the same pair of candidate edges. The specific steps include steps 4-7: Step 4-7: For multiple parallel thin straight line trajectories, the set of initial base point coordinates of all candidate edges in the t-th edge graph Gr _t = <E _t ,E _t+1 > is respectively B _t ,B _t+1 , t=1,2,...,T-1, then the initial base point coordinates of the candidate edges must satisfy the following conditions: Where _Dt is the fixed distance between the edges of the t-th pair of straight line tracks, and _Tt is the distance deviation threshold between the edges of the t-th pair of straight line tracks. For the initial base point coordinate set _B′t , B′t ₊₁ , t＝1, 2, ..., T-1 that meets the above conditions, the effective initial base point coordinates are The corresponding candidate edge is a straight line trajectory edge that meets the requirements; For multiple parallel thick straight line trajectories, the tth pair of edge graphs and The sets of initial base point coordinates of all candidate edges in are Then the initial base point coordinates of the candidate edge must meet the following conditions: in and are the fixed distances between the left and right edges of the t-th pair of straight line tracks, T _t ^L and T _t ^R are the distance deviation thresholds between the left and right edges of the t-th pair of straight line tracks; for the initial base point coordinate set that meets the above conditions The valid initial base point coordinates are The corresponding candidate edge is a straight line trajectory edge that meets the requirements; Step 4-8: For each straight line trajectory edge that meets the requirements, perform straight line fitting on the valid edge pixels contained therein to obtain the straight line equation corresponding to the edge. 5. The method for detecting and tracking a straight line trajectory based on vision according to claim 4, characterized in that: the specific method for pairing the edges of the straight line trajectory is: For multiple parallel thin straight line tracks, the straight line tracks are divided into T straight line tracks from left to right along the horizontal direction of the image, where T represents the number of straight line tracks, and the edge map E is saved as T edge maps E ₁ , E ₂ , ..., E _T according to the straight line tracks to which the edges belong: According to the relative position relationship, two adjacent straight track edges are divided into a pair of Gr _t , and the division method is as follows: Gr _t =<E _t ,E _t+1 >,t＝1,2,...,T-1 For the rough straight line track in the input image, the straight line track edge is divided into three categories: left side, right side and others according to the gradient direction information. The specific steps are as follows: The gradient direction feature GD of each pixel is calculated according to the following formula: Where G _x (x, y) and G _y (x, y) represent the gradient information G _x and G _y of the pixel at the coordinate (x, y) respectively; According to the directional feature GD, all the pixels in the edge map E are divided into three sets: left L, right R, and other O. The division method is as follows: Edge pixels belonging to other O categories will be ignored, that is, if E(x,y)∈O, then E(x,y)＝0; pixels belonging to the left L category are saved as the left edge map ^EL , that is, if E(x,y)∈L, then ^EL (x,y)＝255; pixels belonging to the right R category are saved as the right edge map ^ER , that is, if E(x,y)∈R, then ^ER (x,y)＝255; For multiple parallel thick straight line tracks, divide the straight line tracks into T straight line tracks from left to right along the horizontal direction of the image, where T represents the number of straight line tracks, and save the edge map E as 2T edge maps according to the straight line track and edge category to which the edge belongs. According to the relative position relationship, the edges of the same category of two adjacent straight line trajectories are divided into a pair or The division is as follows: 6. The method for detecting and tracking a straight line trajectory based on vision according to claim 5, characterized in that: the effective pixel number threshold _TP in steps 4-4 and 4-5 is adaptively adjusted according to the straight line trajectory detection result and the number of searches: Where N represents the preset number of searches, P _x represents the sum of the statistical values of valid edge pixels within the search window, and Wherein, _Hi represents the statistical value corresponding to the coordinate i in the statistical histogram, and x represents the horizontal coordinate value of the initial base point. 7. The method for detecting and tracking a straight line trajectory based on vision according to claim 6, characterized in that step S5 specifically comprises the following steps: Step 5-1: The number of valid windows described in step 4-6 reflects the reliability of the current candidate edge detection. The more valid windows there are, the higher the confidence of the candidate edge detection. For thin straight line trajectories, the detection confidence is calculated as: Where c _t represents the detection confidence corresponding to the tth straight line track edge that meets the requirements, _nt represents the number of valid windows of the edge, N represents the preset number of searches, β represents the correction coefficient, and T represents the total number of straight line tracks that meet the requirements; For a rough straight line trajectory, the detection confidence is calculated as: in and They represent the detection confidence corresponding to the left and right edges of the tth straight line trajectory that meets the requirements, and They represent the number of valid windows on the left and right edges of the tth straight line trajectory that meets the requirements, N represents the preset number of searches, β represents the correction coefficient, and T represents the total number of straight line trajectories that meet the requirements; Step 5-2: For thin straight line trajectories, the straight line where the edge of each straight line trajectory that meets the requirements is located and the upper and lower boundaries of the region of interest form two intersection points, and their corresponding confidence levels are and and Then construct the correction coefficient matrix according to the confidence level For the rough straight line trajectory, the straight lines where the left and right edges of each straight line trajectory that meets the requirements are located and the upper and lower boundaries of the region of interest form four intersection points, and their corresponding confidence levels are and and t＝1,2,...,T then construct the correction coefficient matrix according to the confidence level Step 5-3: Calculate the corrected measurement error covariance matrix R' R' = diag(C _k )R Where diag(·) represents the diagonalization operation, R represents the initial measurement error covariance matrix, R = s _R × U, s _R represents the scaling factor, and U represents the unit matrix; Step 5-4: The modified Kalman gain K _k is K _k =P _k M ^T (MP _k M ^T +R') ^-1 Where _Pk represents the covariance matrix of the prior estimation error of the k-th frame image, initialized to _P0 = U, M represents the measurement matrix, for a thin straight line trajectory, For a rough straight line trajectory, The element _Mij in the i-th row and j-th column is as follows Step 5-5: The calculation method of the covariance matrix P _k of the prior estimation error in step 5-4 is: Where Q represents the covariance matrix of process noise, Q = s _Q × U, s _Q represents the scaling factor, the superscript T represents the transpose of the matrix, and A represents the transfer state matrix. For thin straight lines: For a rough straight line trajectory: Represents the covariance matrix of the posterior estimation error of the k-1th frame, and its update process is as follows Where U represents the identity matrix; Step 5-6: For the thin straight line trajectory, the straight line where the edge of the tth straight line trajectory that meets the requirements obtained in step 4-8 is located forms two intersection points with the upper and lower boundaries of the region of interest. The horizontal coordinates of the intersection points are and Construct the k-th frame measurement matrix For the rough straight line trajectory, the straight lines where the left and right edges of the tth straight line trajectory that meets the requirements obtained in steps 4-8 and the upper and lower boundaries of the region of interest form four intersection points, and the horizontal coordinates of the intersection points are and Construct the k-th frame measurement matrix The k-th frame measurement matrix Z _k is calculated as follows: Z _k =MX _k Where _Xk represents the state estimation matrix of the kth frame; Steps 5-7: For thin straight tracks: in and They respectively represent the estimated values of the horizontal coordinates of the two intersection points formed by the straight line where the edge of the t-th straight line trajectory is located and the upper and lower boundaries of the region of interest, which are estimated from the prediction result of the previous frame image; Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between For thick straight line tracks, the width of the straight line track will be used as an additional constraint to track the straight line track: in and The estimated values of the horizontal coordinates of the four intersection points formed by the straight lines where the left and right edges of the t-th straight line trajectory are located and the upper and lower boundaries of the region of interest are estimated from the prediction results of the previous frame image; Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates two adjacent frames The amount of change between Indicates the width of the top of the straight line track, Indicates the change in width between two adjacent frames. Indicates the width of the bottom of the straight line track, Indicates the change in width between two adjacent frames; Step 5-8: The k-th frame state estimation matrix X _k is calculated as follows: in Represents the posterior state estimate matrix of the k-1th frame, which is calculated as follows: For thin straight line trajectories, the initial value is the detection result of the horizontal coordinate of the edge endpoint of the straight line trajectory of the first frame image; for the rough straight line trajectory, the initial value is the detection result of the horizontal coordinates of the edge endpoints of the straight line track and the width of the straight line track of the first frame image; Step 5-9: The final output of the straight line trajectory tracking is The calculation is as follows: