CN113103957B - A blind spot monitoring method, device, electronic equipment and storage medium - Google Patents

Abstract

The disclosure provides a blind area monitoring method, a device, an electronic device and a storage medium, wherein the blind area monitoring method comprises the following steps: acquiring a current frame monitoring image acquired by acquisition equipment on a target vehicle; performing object detection on the current frame monitoring image to obtain type information and position of an object included in the current frame monitoring image; determining a target object positioned in a visual field blind area of the target vehicle according to the position of the object and the visual field blind area of the target vehicle; and generating a monitoring result according to the type information and the position of the target object and the driving state of the target vehicle.

Description

A blind spot monitoring method, device, electronic equipment and storage medium

technical field

The present disclosure relates to the technical field of image processing, and in particular, to a blind spot monitoring method, device, electronic equipment, and storage medium.

Background technique

During the driving process of vehicles and robots, due to the limited range that can be observed, driving blind spots are prone to appear. Due to the existence of blind spots, it is very easy to cause judgment and operation errors of the driver or the self-driving vehicle, reducing the safety of driving.

Contents of the invention

Embodiments of the present disclosure at least provide a blind spot monitoring method, device, electronic equipment, and storage medium.

In the first aspect, an embodiment of the present disclosure provides a blind spot monitoring method, including: acquiring the current frame monitoring image collected by the acquisition device on the target vehicle; performing object detection on the current frame monitoring image to obtain the current frame monitoring The type information and position of the object included in the image; according to the position of the object and the blind spot of the target vehicle, determine the target object located in the blind spot of the target vehicle; according to the type information and position of the target object and the driving state of the target vehicle to generate a monitoring result.

In the embodiment of the present disclosure, by acquiring the current frame monitoring image collected by the acquisition device on the target vehicle, and performing object detection on the current frame monitoring image, determining the type information and position of the object included in the current frame monitoring image, and then according to the position of the object and the blind spot of the target vehicle to determine the target object located in the blind spot of the target vehicle; and then generate monitoring results according to the type information and position of the target object, as well as the driving state of the target vehicle. In this way, different monitoring results can be generated for different types of target objects, thereby improving driving safety and blind spot monitoring performance.

In a possible implementation manner, the monitoring result includes warning information, the driving state of the target vehicle includes steering information of the target vehicle; generating monitoring results, including: determining the level of warning information according to the type information and position of the target object and the steering information of the target vehicle; generating and prompting warning information of the determined level.

In a possible implementation manner, the monitoring result includes vehicle control instructions, the driving state of the target vehicle includes steering information of the target vehicle; The driving state of the vehicle, generating monitoring results, including: generating the vehicle control instruction according to the type information and position of the target object and the steering information of the target vehicle; the blind spot monitoring method also includes: based on the vehicle control instruction to control the target vehicle to travel.

In this way, more targeted and accurate monitoring results can be generated according to the type information and location of the target object and the driving state of the target vehicle.

In a possible implementation manner, determining the target object located in the blind spot of the target vehicle according to the position of the object and the blind spot of the target vehicle includes: The position of the object is to determine the current first distance information between the target vehicle and the object in the current frame monitoring image; according to the current first distance information, determine the target object located in the blind spot of the target vehicle.

In this way, the target object located in the blind spot of the target vehicle's field of view can be accurately detected from all objects included in the monitoring image of the current frame.

In a possible implementation manner, the determining the current first distance information between the target vehicle and the object in the current frame monitoring image according to the position of the object in the current frame monitoring image includes: Based on the current frame monitoring image, determine the distance to be adjusted between the target vehicle and the object in the current frame monitoring image; Scale change information between scales in the frame monitoring images, and historical first distance information between the object and the target vehicle in each frame of the historical frame monitoring images in the multiple frames of historical frame monitoring images, for all The distance information to be adjusted is adjusted to obtain the current first distance information between the target vehicle and the object.

In a possible implementation manner, the adjusting the distance information to be adjusted to obtain the current first distance information between the target vehicle and the object includes: adjusting the distance information to be adjusted , until the error amount of the scale change information is minimum, and the adjusted distance information is obtained; wherein, the error amount is based on the distance information to be adjusted, the scale change information, and each frame in the multi-frame historical frame monitoring image Determining historical first distance information corresponding to historical frame monitoring images; determining the current first distance information based on the adjusted distance information.

In the embodiment of the present disclosure, by continuously optimizing the scale change information of the object between the scales of the current frame monitoring image and the historical frame monitoring images adjacent to the current frame monitoring image, the obtained object can be reduced The error of the scale change information between the image and the scale in the historical frame monitoring image adjacent to the current frame monitoring image, thereby improving the stability of the adjusted distance information.

In a possible implementation manner, before determining the current first distance information based on the adjusted distance information, the blind spot monitoring method further includes: performing target detection on the current frame monitoring image, and determining The position information of the detection frame of the object contained in the current frame monitoring image; based on the position information of the detection frame and the calibration parameters of the acquisition device, determine the current second distance information; the adjustment based on the Determining the current first distance information includes: based on the current second distance information, each frame of the multiple historical frame monitoring images in the multiple historical frame monitoring images, the object and the target vehicle Between the historical second distance information, the historical first distance information corresponding to the frame historical frame monitoring image, and the adjusted distance information, determine the distance offset information for the adjusted distance information; based on the The distance offset information is used to adjust the adjusted distance information to obtain the current first distance information.

In the embodiment of the present disclosure, after the distance offset information is obtained, the adjusted distance information may be further adjusted, so as to obtain the current distance information with high accuracy between the target vehicle and the object.

In a possible implementation manner, the historical second distance between the object and the target vehicle in each frame of the multi-frame historical frame monitoring image based on the current second distance information is Two distance information, the historical first distance information corresponding to the historical frame monitoring image of the frame, and the adjusted distance information, and determining distance offset information for the adjusted distance information includes: based on the current first distance information The second distance information and the historical second distance information corresponding to each frame of the historical frame monitoring image in the multiple frames of the historical frame monitoring image determine the distance corresponding to each frame of the historical frame monitoring image in the multiple frames of the historical frame monitoring image The first linear fitting coefficient of the first fitting curve fitted by the historical second distance information and the current second distance information; each frame of the historical frame monitoring image based on the multiple frames of historical frame monitoring images corresponds to The historical first distance information and the adjusted distance information, determine the historical first distance information and the adjusted historical first distance information corresponding to each frame of the historical frame monitoring image in the multi-frame historical frame monitoring image The second linear fitting coefficient of the second fitting curve fitted by the distance information; based on the first linear fitting coefficient and the second linear fitting coefficient, determine the distance deviation for the adjusted distance information configuration information.

In a possible implementation manner, the determining the current second distance information based on the position information of the detection frame and the calibration parameters of the acquisition device includes: obtaining the current second distance information based on the position information of the detection frame The pixel coordinate value of the corner point set in the detection frame; based on the pixel coordinate value of the set corner point, the calibration parameter of the acquisition device, and the lane line vanishing point used when determining the calibration parameter of the acquisition device The pixel coordinate value of is used to determine the current second distance information.

In a possible implementation manner, the calibration parameters of the collection device include a first height value of the collection device relative to the ground and a focal length of the collection device; the pixel coordinate value based on the set corner point , the calibration parameters of the collection device and the pixel coordinate values of the vanishing point of the lane line used when determining the calibration parameters of the collection device, and determining the current second distance information includes: based on the pixel of the vanishing point of the lane line The coordinate value and the pixel coordinate value of the set corner point in the detection frame determine the first pixel height value of the acquisition device relative to the ground; based on the pixel coordinate value of the set corner point, determine the current frame monitoring A second pixel height value of the object relative to the ground in the image; based on the first pixel height value, the second pixel height value and the first height value, determine the second pixel height value of the object relative to the ground A height value: determining the current second distance information based on the second height value, the focal length of the acquisition device, and the second pixel height value.

In the embodiment of the present disclosure, when the complete detection frame of the object in the current frame monitoring image can be detected, the actual height of the object can be quickly and accurately obtained by introducing the pixel coordinate value of the vanishing point of the lane line and the calibration parameters of the acquisition device value, and can further quickly and accurately determine the current second distance information between the target vehicle and the object.

In a possible implementation manner, the determining the to-be-adjusted distance information between the target vehicle and the object in the current frame monitoring image based on the current frame monitoring image includes: Scale change information between the scale in the frame monitoring image and the scale in the historical frame monitoring image adjacent to the current frame monitoring image; based on the scale change information and the adjacent to the current frame monitoring image The historical first distance information corresponding to the historical frame monitoring image is used to determine the distance information to be adjusted.

In the embodiment of the present disclosure, the historical first distance information with higher accuracy corresponding to the historical frame monitoring image adjacent to the current frame monitoring image, and the distance between the current frame monitoring image and the current frame monitoring image of the object adjacent to the current frame monitoring image The scale change information between the scales in the historical frame monitoring image can obtain the distance information to be adjusted more accurately, so that the adjustment speed can be increased when the current first distance information is determined based on the distance information to be adjusted later.

In a possible implementation manner, the scale change information between the scales of the object in two adjacent frames of monitoring images is determined in the following manner: respectively extracting a plurality of feature points included in the object in the two adjacent frames The first position information in the previous frame monitoring image in the frame monitoring image, and the second position information in the following frame monitoring image; based on the first position information and the second position information, it is determined that the object is in Scale change information between scales in two adjacent frames of monitoring images.

In a possible implementation manner, the determining the scale change information of the object between the scales of two adjacent frames of monitoring images based on the first position information and the second position information includes: based on The first position information is to determine the first scale value of the target line segment formed by the multiple feature points contained in the object in the previous frame of the monitoring image; based on the second position information, determine the target line segment The second scale value in the monitoring image of the next frame; based on the first scale value and the second scale value, determine the scale change information of the object between the scales in two adjacent monitoring images .

In the embodiment of the present disclosure, by extracting the location information of multiple feature points contained in the object in the monitoring image, the location information of the object in the monitoring image can be represented more accurately, so as to obtain more accurate scale change information, which is convenient for the When the change information adjusts the distance information to be adjusted, more accurate current first distance information can be obtained.

In a second aspect, an embodiment of the present disclosure provides a blind spot monitoring device, including:

An acquisition module, configured to acquire the current frame monitoring image collected by the acquisition device on the target vehicle;

A detection module, configured to perform object detection on the current frame monitoring image to obtain type information and positions of objects included in the image;

A determining module, configured to determine the target object located in the blind spot of the target vehicle according to the position of the object and the blind spot of the target vehicle;

A generating module, configured to generate monitoring results according to the type information and location of the target object and the driving state of the target vehicle.

In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processing The processor communicates with the memory through a bus, and when the machine-readable instructions are executed by the processor, the steps of the blind spot monitoring method as described in the first aspect are executed.

In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the blind spot monitoring method as described in the first aspect is executed. step.

In order to make the above-mentioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. The accompanying drawings here are incorporated into the specification and constitute a part of the specification. The drawings show the embodiments consistent with the present disclosure, and are used together with the description to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. For those skilled in the art, they can also make From these drawings other related drawings are obtained.

FIG. 1 shows a flow chart of a blind spot monitoring method provided by an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of determining a blind area of vision provided by an embodiment of the present disclosure;

FIG. 3 shows a flow chart of a specific method for determining current first distance information provided by an embodiment of the present disclosure;

FIG. 4 shows a flowchart of a method for determining scale change information provided by an embodiment of the present disclosure;

FIG. 5 shows a flow chart of a method for determining distance information to be adjusted provided by an embodiment of the present disclosure;

FIG. 6 shows a flowchart of a method for determining current first distance information provided by an embodiment of the present disclosure;

FIG. 7 shows a flowchart of a method for determining current second distance information provided by an embodiment of the present disclosure;

FIG. 8 shows a schematic diagram of the positional relationship between a target device, a collection device, and a target object provided by an embodiment of the present disclosure;

FIG. 9 shows a schematic diagram of a detection frame of a target object provided by an embodiment of the present disclosure;

Fig. 10 shows a schematic diagram of the principle of determining the current second distance information provided by an embodiment of the present disclosure;

Fig. 11 shows a schematic diagram of a scenario for determining the current second distance information provided by an embodiment of the present disclosure;

Fig. 12 shows a schematic structural diagram of a blind spot monitoring device provided by an embodiment of the present disclosure;

Fig. 13 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only It is a part of the embodiments of the present disclosure, but not all of them. The components of the disclosed embodiments generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort shall fall within the protection scope of the present disclosure.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The term "and/or" in this article only describes an association relationship, which means that there can be three kinds of relationships, for example, A and/or B can mean: there is A alone, A and B exist at the same time, and B exists alone. situation. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

After research, it is found that when using radar to monitor the target object in the blind area, since the point cloud points obtained by the radar during scanning are aimed at the entire range of the blind area, in addition to paying attention to vehicles, pedestrians and signage during monitoring, it will also monitor to other non-concerned objects. When blind spot monitoring is carried out in this way, once an object is detected by the radar in the blind spot of the vehicle, an alarm will be generated. But in fact, not all objects will affect the driving safety of vehicles when they are located in the blind area of vision, which leads to many invalid alarms, resulting in the problem of poor monitoring performance in the current blind area monitoring method.

Based on the above research, the embodiments of the present disclosure provide a blind spot monitoring method. The embodiments of the present disclosure obtain the current frame monitoring image collected by the acquisition device on the target vehicle, and perform object detection on the current frame monitoring image to determine the current frame monitoring image. The type information and position of the object included in the image, and then according to the position of the object and the blind spot of the target vehicle, judge the target object located in the blind spot of the target vehicle; then according to the type information and position of the target object, and the driving of the target vehicle Status, generate monitoring results. In this way, different monitoring results can be generated for different types of target objects, thereby improving blind spot monitoring performance and driving safety.

In order to facilitate the understanding of this embodiment, a blind spot monitoring method disclosed in the embodiments of the present disclosure is first introduced in detail. The execution subject of the blind spot monitoring method provided in the embodiments of the present disclosure is generally a computer device with a certain computing power. The computer The device includes, for example: a terminal device or a server or other processing device, and the terminal device may be a computing device, a vehicle-mounted device, or the like. In some possible implementation manners, the blind spot monitoring method may be implemented by a processor invoking computer-readable instructions stored in a memory.

Referring to FIG. 1 , which is a flow chart of a blind spot monitoring method provided by an embodiment of the present disclosure, the blind spot monitoring method includes the following S101-S104:

S101, acquiring the current frame monitoring image collected by the collection device on the target vehicle;

S102. Perform object detection on the current frame monitoring image to obtain type information and positions of objects included in the current frame monitoring image;

S103. Determine the target object located in the blind spot of the target vehicle according to the position of the object and the blind spot of the target vehicle;

S104. Generate a monitoring result according to the type information and location of the target object and the driving state of the target vehicle.

For the above S101, in different scenarios, the corresponding target vehicles are also different.

Exemplarily, in the scenario where the driver drives the vehicle, the target vehicle may include, for example, a vehicle driven by the driver; in the scenario of automatic driving, the target vehicle may include, for example, an autonomous vehicle; Cargo robots can be included. Embodiments of the present disclosure are described by taking the target vehicle as an example.

A collection device may also be mounted on the target vehicle, and the collection device may be a monocular camera installed on the target vehicle for shooting while the target vehicle is driving. Exemplarily, if the target area includes: a blind spot of the vehicle, the collection device may be installed on a column of the vehicle, and the photographing lens of the collection device faces the blind spot of the vehicle.

Among them, different target vehicles may have different corresponding blind spots due to their different models. The embodiments of the present disclosure refer to the national standard to determine the blind area of the field of view. For details, see FIG. 2 , which is a schematic diagram of determining the blind area of the field of view provided by the embodiment of the present disclosure. In FIG. 2 , a vehicle 1 is equipped with a collection device 2 , and the target area collected by the collection device 2 includes the blind spots in the field of vision including the positions indicated by 3 and 4 .

Specifically, when acquiring the current frame monitoring image collected by the acquisition device on the target vehicle, for example, the following method can be adopted: obtain the monitoring video obtained by the acquisition device on the target vehicle from image acquisition of the target area; determine the current frame from the monitoring video A frame monitoring image; wherein, the target area includes: an area within the field of view of the acquisition device.

When the acquisition equipment is used to photograph the target area, the shooting direction can be set in advance; after the acquisition equipment is mounted on the target vehicle, it can be determined that the target area to be photographed is the area within the field of view of the acquisition equipment. When the vehicle is running or stopping, the acquisition device can collect images of the target area, obtain the monitoring video, and determine the current frame monitoring image from the monitoring video.

Wherein, when using the monitoring video to determine the monitoring image of the current frame, it can be determined by determining the video frame image from the monitoring video, for example, taking the frame closest to the current time among the captured video frame images as the monitoring image of the current frame.

With respect to the above S102, after the current frame monitoring image is collected by the above S101, object detection may also be performed on the current frame monitoring image to obtain the type information and position of the object included in the current frame monitoring image.

In a specific implementation, when performing object detection on the current frame monitoring image and determining the type information of the object included in the image, for example, a pre-trained target detection neural network can be used to perform object detection (object detection) processing on the current frame monitoring image, Exemplarily, when performing object detection on the current frame monitoring image, at least one of the following object detection algorithms can be used: convolutional neural network (Convolutional Neural Networks, CNN), target detection network (Region-based CNN, RCNN) , Fast Neural Network (Fast RCNN) and Faster Neural Network (Faster RCNN).

When the object detection algorithm is used to monitor the current frame of the monitoring graphics, the objects that can be detected include, for example, other driving vehicles, pedestrians, road facilities, and road obstacles.

When performing object detection on the monitoring image of the current frame, the position of the object included in the image can also be obtained. Through the detected position of the object in the image, the actual position of the object during the actual driving of the target vehicle can be further determined.

For the above S103, using the position of the object and the blind spot of the target vehicle, the following method can be used to determine the target object located in the blind spot of the target vehicle: according to the position of the object in the current frame monitoring image, determine the target vehicle and the current frame monitoring image The current first distance information of the object in the center; according to the current first distance information, determine the target object located in the blind spot of the target vehicle.

At present, when using the image collected by the monocular camera to determine the distance, the monocular camera mounted on the smart car will have problems such as road bumps or obstructions due to changes in driving conditions during the driving of the smart car. Under certain circumstances, when performing distance measurement based on the detection frame corresponding to the object in the current frame monitoring image, the accurate distance to the object may not be detected. For example, due to the bumpy road surface of the acquisition device, the size of the detection frame collected in the monitoring image is unstable. , so when the distance between the smart car and the object is continuously detected based on the detection frame, the obtained distance between the smart car and the object is not stable in timing.

In order to use the image collected by the monocular camera to detect the distance between the target vehicle and the object as accurately and stably as possible, the embodiment of the present disclosure also proposes a distance detection scheme,

Referring to FIG. 3 , it is a flow chart of a specific method for determining the current first distance information provided by an embodiment of the present disclosure. The method includes the following S301-S302:

S301. Determine distance information to be adjusted between the target vehicle and an object in the current frame of the monitoring image based on the current frame of the monitoring image.

Exemplarily, the object may include, but not limited to, a vehicle, a pedestrian, a fixed obstacle, etc., and the present disclosure is introduced by taking the object as an example in the embodiment.

Exemplarily, the current frame monitoring images provided by the embodiments of the present disclosure are all monitoring images in which an object is not detected for the first time. If the current frame monitoring image is a monitoring image in which an object is detected for the first time, it can be directly based on the position of the object in the current frame monitoring image. Position information, as well as the parameter information of the acquisition device obtained in the above calibration process and the pixel coordinate value of the vanishing point determine the current second distance information between the object and the current second distance information, and the current second distance information can be directly used as the current first distance information, specifically The process of determining the current second distance information is described in detail later.

Exemplarily, in the case that the current frame monitoring image is not an object collected for the first time, the current first distance information corresponding to the current frame monitoring image, or the historical first distance information corresponding to each frame of historical frame monitoring image all represent the adjusted The obtained distance information.

Exemplarily, when determining the distance information to be adjusted between the target vehicle and the object based on the current frame monitoring image, it may be based on the historical first distance information corresponding to the historical frame monitoring image adjacent to the current frame monitoring image, and the current frame monitoring image. The scale change information between the image and the scales in the historical frame monitoring image adjacent to the current frame monitoring image is determined, and then the to-be-adjusted distance information is adjusted subsequently.

S302, based on the scale change information of the object in the multi-frame historical frame monitoring image collected by the acquisition device between the scales in two adjacent frames of the monitoring image, and the relationship between the object in each frame of the multi-frame historical frame monitoring image and the object in the historical frame monitoring image The historical first distance information between the target vehicles is adjusted to the distance information to be adjusted to obtain the current first distance information between the target vehicle and the object.

Exemplarily, the scale change information of the object in two adjacent frames of monitoring images (such as including monitoring image i and monitoring image j) in the multiple frames of historical frame monitoring images collected by the acquisition device includes the object's scale change information in the next frame of monitoring image j. The ratio of the scale to the scale of the object in the previous frame of monitoring image i, the specific determination process will be described later.

Exemplarily, the embodiment of the present disclosure determines the historical first distance information between the target vehicle and the object corresponding to each frame of the historical frame monitoring image in the same manner as the method of determining the current first distance information between the target vehicle and the object, so The embodiment of the present disclosure will not repeat the process of determining the historical first distance information.

In the embodiment of the present disclosure, it can be based on the scale change information of the object in two adjacent frames of monitoring images in multiple frames of historical frame monitoring images, and the historical first distance information between the target vehicle and the object that has been adjusted in the historical process , adjust the distance information to be adjusted based on the current monitoring image, so that the distance between the target vehicle and the object corresponding to two adjacent frames of monitoring images can be changed more smoothly, and can truly reflect the distance between the target vehicle and the object during driving. The actual distance changes can improve the stability of the predicted distance between the target vehicle and the object in time series.

In addition, the scale change information of the object in two adjacent frames of monitoring images can also reflect the distance change between the target vehicle and the object. In the multiple frames of historical frame monitoring images, each frame of historical frame monitoring images corresponds to the historical distance between the target vehicle and the object. The first distance information is the relatively accurate distance information obtained after adjustment. Therefore, in the multi-frame historical frame monitoring images collected by the acquisition device based on the object, the scale change information in two adjacent frames of monitoring images and the multi-frame historical frame monitoring images After the historical first distance information between the target vehicle and the object corresponding to each frame of the historical frame monitoring image is adjusted, more accurate current first distance information can be obtained.

In the embodiment of the present disclosure, when determining the current first distance information, it can be based on the scale change information between the scales of the object in two adjacent frames of monitoring images in multiple frames of historical frame monitoring images, and the information that has been adjusted in the historical process The historical first distance information between the target vehicle and the object is obtained, and the distance information to be adjusted based on the current monitoring image is adjusted, so that the distance between the same object and the target vehicle in two adjacent monitoring images can change It is relatively stable, can truly reflect the actual distance change between the target vehicle and the object during driving, and can improve the stability of the predicted distance between the target vehicle and the object in time series.

First, for the scale change information mentioned above, as shown in Figure 4, the scale change information between the scales of the object in two adjacent frames of monitoring images can be determined in the following manner, including the following S401-S402:

S401. Extract first position information of a plurality of feature points included in the object in a previous frame of monitoring image and second position information of a subsequent frame of monitoring image in two adjacent frames of monitoring images.

Exemplarily, target detection can be performed on the monitoring image based on the pre-trained target detection model, and a detection frame used to represent the position of the object in the monitoring image can be obtained, and then a plurality of feature points constituting the object can be extracted in the detection frame, these A feature point refers to a point in an object where pixels change drastically, such as an inflection point, a corner point, and the like.

S402. Based on the first position information and the second position information, determine scale change information between scales of the object in two adjacent frames of monitoring images.

Exemplarily, the connection of any two feature points in the same frame of the monitoring image can constitute a line segment, so that the first position information of any two feature points in the previous frame of the monitoring image can be obtained The scale of the line segment formed by any two feature points in the previous frame monitoring image, similarly, from the second position information of any two feature points in the next frame monitoring image, the line segment formed by any two feature points can be obtained in The scales in the monitoring image of the next frame can be obtained in this way, respectively, the scales of the multiple line segments on the object in the monitoring image of the previous frame, and the scales of the multiple line segments in the monitoring image of the next frame respectively.

Further, according to the scales of the multiple line segments in the previous frame of the monitoring image and the scales of the multiple line segments in the subsequent frame of the monitoring image, the scale change information of the object in two adjacent frames of monitoring images can be determined.

Specifically, for S402, when determining the scale change information between the scales of the object in two adjacent frames of monitoring images based on the first position information and the second position information, the following S4021-S4023 are included:

S4021. Based on the first position information, determine a first scale value of a target line segment formed by a plurality of feature points included in the object in the previous frame of the monitoring image.

S4022. Based on the second position information, determine a second scale value of the target line segment in the next frame of the monitoring image.

Exemplarily, the target line segment contains n lines, n is greater than or equal to 1, and is smaller than the set threshold, based on the first position information of the feature points contained in each target line segment, the first scale value corresponding to the target line segment can be obtained, And, based on the second position information of the feature points contained in each target line segment, the second scale value corresponding to the target line segment can be obtained.

S4023. Based on the first scale value and the second scale value, determine scale change information between scales of the object in two adjacent frames of monitoring images.

Exemplarily, the ratio of the second scale value corresponding to any target line segment to the first scale value can be used to represent the scale change information corresponding to the target line segment, and further according to the scale change information corresponding to multiple target line segments , determine the scale change information of the object in two adjacent frames of monitoring images, for example, the average value of the scale change information corresponding to the set number of target line segments can be used as the scale change information of the object in two adjacent frames of monitoring images.

Compared with the method of expressing the scale of the object through the position information of the two corner points of the detection frame in the monitoring image, for example, the position information of the upper left corner point and the lower right corner point of the detection frame in the monitoring image indicates that the object is in the monitoring image In the way of scale, the embodiment of the present disclosure determines the scale change information of the object in two adjacent frames of monitoring images by selecting the position information of a plurality of feature points in two adjacent frames of monitoring images. The included position information of multiple feature points in the monitoring image can more accurately represent the position information of the object in the monitoring image, thereby obtaining more accurate scale change information.

In the embodiment of the present disclosure, by extracting the location information of multiple feature points contained in the object in the monitoring image, the location information of the object in the monitoring image can be represented more accurately, so as to obtain more accurate scale change information, which is convenient for the When the change information adjusts the distance information to be adjusted, more accurate current first distance information can be obtained.

For the above S302, when determining the distance information to be adjusted between the target vehicle and the object in the current frame monitoring image based on the current frame monitoring image, as shown in FIG. 5, the following S501-S502 may be included:

S501. Obtain scale change information between the scale of the object in the current frame of the monitored image and the scale of the historical frame of the adjacent monitored image.

Exemplarily, the historical frame monitoring image adjacent to the current frame monitoring image refers to the previous frame monitoring image at the acquisition time before the current frame monitoring image, and the object is in the current frame monitoring image and the current frame monitoring image adjacent to the historical frame The scale change information between monitored images can be represented by the ratio of the scale of the object in the current frame monitored image to the scale of the object in the historical frame monitored image adjacent to the current frame monitored image. The specific determination process will be carried out later elaborate.

S502. Based on the scale change information and the historical first distance information corresponding to the historical frame monitoring image adjacent to the current frame monitoring image, determine distance information to be adjusted.

Considering that the target vehicle and the object are approaching, the scale of the object in the collected monitoring images will gradually increase, that is, the scale of the object in two adjacent monitoring images and the target vehicle and object corresponding to the two frames of monitoring images There is a proportional relationship between the distances. Based on this, the distance information to be adjusted can be determined by the following formula (1):

d _{0_scale} = scale×D _{1_final} ; (1);

Among them, d _{0_scale} represents the distance information to be adjusted; scale represents the ratio of the scale of the object in the current frame monitoring image to the scale of the object in the historical frame monitoring image adjacent to the current frame monitoring image; D _{1_final} represents the current frame monitoring image The historical first distance information corresponding to the adjacent historical frame monitoring images.

In the embodiment of the present disclosure, the historical first distance information corresponding to the historical frame monitoring image adjacent to the current frame monitoring image, and the scale of the object in the current frame monitoring image and the historical frame monitoring image adjacent to the current frame monitoring image The scale change information between them can obtain more accurate distance information to be adjusted, so that when the current first distance information is determined based on the distance information to be adjusted later, the adjustment speed can be increased.

Exemplarily, there may be errors in the acquired scale change information of the object between the current frame monitoring image and the historical frame monitoring images adjacent to the current frame monitoring image, such as shaking when the current monitoring image is taken, or detection The object error, the scale change information obtained based on this will have a sudden change compared with the scale change information in two adjacent frames of monitoring images in multiple frames of historical frame monitoring images, so the distance information to be adjusted based on this is compared with the adjacent There will also be sudden changes in the first distance information in history. At this time, the scale change information in two adjacent frames of monitoring images in the multi-frame historical frame monitoring images collected by the acquisition device, and the scale change information in each of the multi-frame historical frame monitoring images can be used. The historical first distance information corresponding to the frame history frame monitoring image is used to adjust the distance information to be adjusted.

Specifically, when adjusting the distance information to be adjusted to obtain the current first distance information between the target vehicle and the object, as shown in FIG. 6 , the following steps S601-S603 may be included:

S601, adjust the distance information to be adjusted until the error amount of the scale change information is the smallest, and obtain the adjusted distance information; wherein, the error amount is based on the distance information to be adjusted, the scale change information, and each historical frame in the multi-frame historical frame monitoring image The historical first distance information corresponding to the monitoring image is determined.

Exemplarily, the error amount used to represent the scale change information of the object between the current frame monitoring image and the historical frame monitoring images adjacent to the current frame monitoring image can be predicted based on the following formula (2):

Among them, E represents the error amount of the scale change information of the object between the current frame monitoring image and the historical frame monitoring image adjacent to the current frame monitoring image; T includes the frame number of the object’s monitoring image, and T is less than or equal to the preset frame number; t is used to indicate the historical frame monitoring image, representing the tth frame historical frame monitoring image starting from the current frame monitoring image, such as t=1 representing the first frame historical frame monitoring image starting from the current frame monitoring image; L _t represents The preset weight of the tth frame historical frame monitoring image starting from the current frame monitoring image when determining the error amount E, D _{t_final} represents the first distance information in history corresponding to the tth frame historical frame monitoring image starting from the current frame monitoring image; scale _i represents the scale change information between the i-th historical frame monitoring image starting from the current frame monitoring image and the i+1-th historical frame monitoring image.

Exemplarily, the above formula (2) can be optimized through various optimization methods, such as adjusting the d _{0_scale} in the above formula (2) by including but not limited to the Newton gradient descent method. When E is the smallest, we get Adjusted distance information D _{0_scale} .

Through the above method, the scale change information of the object between the current frame monitoring image and the historical frame monitoring image adjacent to the current frame monitoring image is continuously optimized, which can reduce the obtained objects in the current frame monitoring image and the current frame monitoring image. The error of the scale change information between the adjacent historical frames of the monitoring image is monitored, thereby improving the stability of the determined adjusted distance information.

S602. Determine current first distance information based on the adjusted distance information.

Exemplarily, after the adjusted distance information is obtained, in order to further improve the accuracy of the adjusted distance information, the adjusted distance information may be further adjusted to obtain the current first distance information between the target vehicle and the object .

Specifically, before determining the current first distance information based on the adjusted distance information, as shown in FIG. 7 , the blind spot monitoring method provided by the embodiment of the present disclosure further includes the following S701-S702:

S701. Perform target detection on the current frame of monitoring image, and determine position information of a detection frame of an object contained in the current frame of monitoring image.

S702. Determine the current second distance information based on the position information of the detection frame and the calibration parameters of the acquisition device.

Exemplarily, before the target vehicle travels, the acquisition device installed on the target vehicle can be calibrated, for example, the acquisition device is installed on the top of the target vehicle, as shown in Figure 6, so that the target vehicle is located in the middle of the parallel lane line, keeping The optical axis of the acquisition device is parallel to the horizontal ground and parallel to the forward direction of the target vehicle. In this way, the focal length (f _x , f _y ) of the acquisition device and the height H _c of the acquisition device relative to the ground can be obtained.

Exemplarily, a pre-trained target detection model can be used to perform target detection on the current frame monitoring image to obtain the object contained in the current frame monitoring image and the corresponding detection frame of the object, as shown in Figure 8, the position information of the detection frame It can contain the position information of the corner points of the detection frame in the current frame of the monitoring image, for example, it can contain the pixel coordinate values of the corner points A, B, C and D in the current frame of the monitoring image.

Further, according to the principle of pinhole imaging, the following formulas (3) and (4) can be obtained:

Among them, H _x represents the actual width of the object; H _y represents the actual height of the object relative to the ground; w _b represents the pixel width of the object in the current frame monitoring image, which can be determined by the pixel width of the detection frame ABCD of the object; h _b represents The pixel height of the object relative to the ground can be determined by the pixel height of the detection frame ABCD of the object; D ₀ represents the current second distance information between the target vehicle and the object.

Exemplarily, in one embodiment, H _x and H _y can be determined by the type of the detected object. For example, when the object is a vehicle, it can be determined based on the detected type of the target vehicle, and the pre-stored vehicle type and The corresponding height of the vehicle and the corresponding relationship between the width determine the actual width and actual height of the target vehicle.

Exemplarily, the width w _b of the object in the current frame monitoring image can be determined by the pixel coordinate value of the corner point AB in the current frame monitoring image in the detection frame ABCD as shown in Figure 9, or by the corner point CD in the current frame The pixel coordinate value in the monitoring image is determined; the height h _b of the object in the monitoring image of the current frame can be determined by the pixel coordinate value of the corner point BC in the monitoring image of the current frame, or by the pixel of the corner point AD in the monitoring image of the current frame The coordinate values are determined, and details are not described here.

Considering that the type of the object cannot be identified, it may not be possible to directly obtain the actual height or actual width of the object. The embodiment of the present disclosure takes determining the actual height of the object as an example. For the above S702, based on the detection frame When the position information and the calibration parameters of the collection device are determined to determine the current second distance information, the following steps S7021-S7022 are included:

S7021. Based on the position information of the detection frame, acquire the pixel coordinate value of the set corner point in the detection frame.

S7022. Determine the current second distance information based on the pixel coordinate value of the set corner point, the calibration parameter of the acquisition device, and the pixel coordinate value of the vanishing point of the lane line used when determining the calibration parameter of the acquisition device.

The current second distance will be determined based on the pixel coordinate value of the set corner point, the calibration parameter of the collection device, and the pixel coordinate value of the vanishing point of the lane line used when determining the calibration parameter of the collection device, as shown in Figure 10 below. The principle of the information is explained:

For example, in the initial calibration process of the acquisition device, the target vehicle can be parked between the parallel lane lines, and the distant parallel lane lines intersect at one point when the phase plane projection of the acquisition device is projected, which can be called lane line disappearance point, the vanishing point of the lane line approximately coincides with the V point in Figure 10, the projected position of the acquisition device in the monitoring image can be represented by the vanishing point of the lane line, and the pixel coordinate value of the vanishing point of the lane line can indicate that the acquisition device is in the current frame Monitor pixel coordinate values in an image.

As shown in Figure 10, the distance between the two points of EG can represent the actual height H _c of the acquisition device relative to the ground; the distance between the two points of FG can represent the actual height H _y of the object relative to the ground; The distance of can represent the pixel height h _b of the object relative to the ground; the distance between two points of MV can represent the pixel height of the acquisition device relative to the ground.

Further, as shown in Fig. 10, according to the pinhole imaging principle, when the acquisition device captures the current frame monitoring image, the ratio of the actual height H _c of the acquisition device relative to the ground to the actual height H _y of the object relative to the ground is equal to The ratio of the pixel height of the acquisition device relative to the ground to the pixel height h _b of the object relative to the ground, so that after determining the pixel coordinate values of point M, point V and point N, the position of the target object relative to the ground can be further determined The pixel height h _b and the pixel height of the acquisition device relative to the ground are used to predict the actual height H _y of the object relative to the ground.

Further, after the actual height of the object relative to the ground is predicted, the current second distance information may be determined in combination with the above formula (3).

The principle of determining the current second distance information is introduced above in conjunction with FIG. 10 . The following will introduce the specific process of how to determine the current second distance information in conjunction with FIG. 11 :

As shown in Figure 11, after de-distorting the current frame monitoring image, an image coordinate system is established for the current frame monitoring image, and the pixel coordinate value (x _v , y _v ); the pixel coordinate value (x _tl , y _tl ) of the upper left corner point A of the detection frame of the object, and the pixel coordinate value (x _br , y _br ) of the lower right point C, and further, you can pass the corner point AC along the y-axis The pixel coordinate value in the direction determines the distance between the two points MN as shown in Figure 10; the distance between the two points MV as shown in Figure 10 can be determined by the pixel coordinate value of the corner point CV along the y-axis direction.

Specifically, the calibration parameters of the collection device include the first height value of the collection device relative to the ground and the focal length of the collection device; for the above S7022, based on the pixel coordinate value of the set corner point, the calibration parameters of the collection device, and the determination of the collection device The pixel coordinate value of the vanishing point of the lane line used when calibrating the parameters, and when determining the current second distance information, include the following S70221-S70224:

S70221. Based on the pixel coordinate value of the disappearing point of the lane line and the pixel coordinate value of the set corner point in the detection frame, determine a first pixel height value of the collection device relative to the ground.

Combining with what is shown in Figure 11 above, the first pixel height value can be obtained: y _br -y _v .

S70222. Based on the pixel coordinate value of the set corner point, determine a second pixel height value of the object in the current frame monitoring image relative to the ground.

For example, the difference between the pixel coordinate values of the two corner points AC in FIG. 11 along the y-axis can be used as the second pixel height value here, which can be represented by h _b .

S70223. Based on the first pixel height value, the second pixel height value and the first height value, determine a second height value of the object relative to the ground.

Among them, H _c represents the first height value, which is used to represent the actual height of the collection device relative to the ground, which can be obtained when the collection device is calibrated; _Hy represents the second height value, which is used to represent the actual height of the object relative to the ground .

S70224. Determine the current second distance information based on the second height value, the focal length of the acquisition device, and the second pixel height value.

Exemplarily, the current second distance information may be determined by the above formula (3).

In the embodiment of the present disclosure, when the complete detection frame corresponding to the object in the current frame monitoring image can be detected, the actual value of the object can be quickly and accurately obtained by introducing the pixel coordinate value of the vanishing point of the lane line and the calibration parameters of the acquisition device. The height value can further quickly and accurately determine the current second distance information between the target vehicle and the object.

After obtaining the current second distance information between the target vehicle and the object, when determining the current first distance information based on the adjusted distance information in the above S603, the following S6031-S6032 are included:

S6031, based on the current second distance information, the historical second distance information between the object and the target vehicle in each frame of the multi-frame historical frame monitoring image, and the historical first distance information corresponding to the historical frame monitoring image and the adjusted distance information, determining distance offset information for the adjusted distance information.

Exemplarily, the current second distance information and each historical second distance information are the distance information between the target vehicle and the object determined based on a single-frame monitoring image. In this way, when determining the second distance information, if it can be detected The accurate and complete detection frame of the object can be based on the position information of the detection frame to obtain the second distance information with high accuracy between the target vehicle and the object. On the contrary, if the accurate detection frame of the object cannot be detected, or the detected object’s The detection frame is incomplete, and the accuracy of the obtained second distance information is low. Therefore, the accuracy of multiple second distance information determined based on this method is high, but fluctuates greatly.

Exemplarily, each historical first distance information is distance information determined based on multiple frames of monitoring images, and the adjusted distance information is also adjusted distance information based on multiple historical first distance information. Therefore, based on this method The fluctuations between the obtained multiple historical first distance information and the adjusted distance information are small, but when determining the historical first distance information and the adjusted distance information, the scale changes corresponding to two adjacent frames of monitoring images are used information, and the determination process of the scale change information depends on the position information of the feature points of the recognized object in the monitoring image. When there is an error, the error will be accumulated. Therefore, the determined multiple historical first distance information and the adjusted distance information The accuracy of is compared with the accuracy of the second distance information determined based on the complete detection frame.

Considering the high accuracy of the current second distance information determined based on the detection frame and the historical second distance information, and the high stability between the historical first distance information determined based on the scale change information and the adjusted distance information, in order to obtain For the current first distance information with high accuracy and stability, the adjusted distance information can be further adjusted to the determined distance information between the target vehicle and the object corresponding to the determined multi-frame monitoring images in two ways.

S6032. Adjust the adjusted distance information based on the distance offset information to obtain the current first distance information.

Exemplarily, after the distance offset information is obtained, the adjusted distance information may be further adjusted based on the distance offset information, so that the current first distance information is more accurate.

In the embodiment of the present disclosure, after the distance offset information is obtained, the adjusted distance information may be further adjusted, so as to obtain the current distance information with high accuracy between the target vehicle and the object.

In one embodiment, based on the current second distance information, the historical second distance information between the object and the target vehicle in each frame of the multiple frames of historical frame monitoring images, the frame of historical frame monitoring images corresponds to When determining the distance offset information for the adjusted distance information, the following steps S60311-S60313 may be included:

S60311, based on the current second distance information and the historical second distance information corresponding to each frame of the historical frame monitoring image in the multi-frame historical frame monitoring image, determine the history corresponding to each frame of the historical frame monitoring image in the multi-frame historical frame monitoring image The first linear fitting coefficient of the first fitting curve formed by fitting the second distance information and the current second distance information.

Exemplarily _, the current second distance information can be represented _by D ₀ , and a plurality of historical second distance information can be represented by D ₁ , _{D 2 , D 3 .} Linear fitting is obtained by a plurality of historical second distance information and the first fitting curve formed by the current second distance information, the first fitting curve can be represented by the following formula (6):

y ₁ =ax+bx ² +c (6);

In the fitting process, the frame numbers 0, 1, 2, 3... of the monitoring images used when determining multiple second distance information can be used as x values, and the second distance information D ₀ , D corresponding to the frame numbers respectively ₁ , D ₂ , D ₃ . . . input formula (6) as the y value, and the first linear fitting coefficients: a, b, c can be obtained.

S60312, based on the historical first distance information corresponding to each frame of the historical frame monitoring image in the multiple frames of historical frame monitoring images and the adjusted distance information, determine the history corresponding to each frame of the historical frame monitoring images in the multiple frames of historical frame monitoring images A second linear fitting coefficient of a second fitting curve formed by fitting the first distance information and the adjusted distance information.

Exemplarily, the adjusted distance information can be represented by D _{0_scale} , and multiple historical first distance information _{can be represented by D 1_final , D 2_final ,} D _{3_final .} Linear fitting, obtains the second fitting curve that is made of multiple historical first distance information and adjusted distance information, this second fitting curve can be represented by following formula (7):

y ₂ =a'x+b'x ² +c'(7);

In the fitting process, the frame numbers 0, 1, 2, 3... of the monitoring images used when determining multiple historical first distance information and adjusted distance information can be used as x values, and the adjustments corresponding to the frame numbers respectively The last distance information and multiple historical first distance information D _{0_scale} , D _{1_final} , D _{2_final} , D _{3_final} . .

S60313. Determine distance offset information for the adjusted distance information based on the first linear fitting coefficient and the second linear fitting coefficient.

Exemplarily, the distance bias information can be determined by the following formula (8):

L=(a/a'+b/b'+c/c')/3 (8);

The distance offset information determined in this way can be adjusted according to the following formula (9) to adjust the distance information to be adjusted to obtain the current first distance information D _{0_final} :

D _{0_final} = D _{0_scale} × L (9);

In another embodiment, based on the current second distance information, the historical second distance information between the object and the target vehicle in each frame of the multi-frame historical frame monitoring image, the historical frame monitoring image of the frame The corresponding historical first distance information and the adjusted distance information, when determining the distance offset information for the adjusted distance information, can also be determined by the Kalman filter algorithm, and further determine the current first distance based on the Kalman filter algorithm information.

When determining the current first distance information based on the Kalman filter algorithm, it can be determined by the following formula (10):

D _{0_final} =kal(D _{0_scale} ,D ₀ ,R,Q) (10);

Among them, R represents the variance of D _{0_scale} and D _{1_final} , D _{2_final} , D _{3_final} ...; Q represents the variance of D ₀ and D ₁ , D ₂ , D ₃ ..., and the distance bias information for D _{0_scale} can be determined through R and Q , further correcting the adjusted distance information based on the distance offset information to obtain the current first distance information with high accuracy.

After the current first distance information is determined, the target object located in the blind spot of the target vehicle can be determined according to the current first distance information between each object in the current frame image and the target vehicle.

Exemplarily, according to the manner of determining the blind spot of the target vehicle shown in FIG. 2 above, the blind spot of the target vehicle can be determined. In addition, using the current first distance information and the determined blind spot of the target vehicle, the detected object falling into the blind spot of the visual field, that is, the target object, can be determined.

Here, the target object may include, for example, other driving vehicles, pedestrians, road facilities, and parts of road obstacles among the above objects.

Regarding the above S104, the monitoring result can be generated according to the type information and location of the target object, and the driving state of the target vehicle.

Here, since the generated monitoring results are determined based on images, when the monitoring results are used to guide the self-driving vehicle or assist the driver in driving, it is usually necessary to continuously generate the monitoring results for a continuous period of time. During this continuous time, multiple frames of images will be acquired, but the multiple frames of images are discrete relative to this continuous time. In order to obtain a more accurate monitoring of the target object, it is also possible to perform tracking and smoothing processing on continuous frame images during object monitoring, for example, by using interpolation to further improve accuracy.

In addition, since the distance between the target object and the target vehicle in continuous time can be determined using multiple frames of monitoring images corresponding to discrete time by using the tracking smoothing method, this method can also alleviate the need for acquisition equipment to quickly and continuously acquire multiple frames of monitoring images. pressure and reduce equipment wear and tear.

Exemplarily, in order to make the obtained monitoring results less discrete, to ensure the safety of the target vehicle while driving, it is correspondingly necessary to obtain a frame of monitoring image in 0.1 seconds; if a frame of monitoring image is obtained in 0.5 seconds, in In a fast-speed scene, a sudden impact may occur within 0.5 seconds, which means that safety cannot be guaranteed. However, the frequency of acquiring a frame of monitoring images in 0.1 seconds requires more power consumption for acquisition equipment than the frequency of acquiring a frame of monitoring images in 0.2 seconds; Within 0.1 second, a relatively accurate frame of predicted monitoring image can guarantee safety.

In addition, after the target object is determined, different target objects at different locations have different influences on the target vehicle. Therefore, it is also possible to determine the Corresponding monitoring results.

Specifically, the monitoring result may include warning information, for example. The driving state of the target vehicle may include steering information of the target vehicle, for example.

In a specific implementation, when generating monitoring results according to the type information and position of the target object and the driving state of the target vehicle, for example, the following method can be adopted: determine the warning information according to the type information and position of the target object and the steering information of the target vehicle level; generate a certain level of alarm information and prompt.

Wherein, the type information of the target object may include, for example, pedestrians. Since the target object is located in the blind spot of the target vehicle, that is, the target vehicle may affect driving safety due to the target object being located in the blind spot of the target vehicle, a monitoring result including warning information can be generated.

In a possible implementation manner, the steering information of the target vehicle indicates that the target vehicle is turning left, and the position of the target object indicates that the target object is in the blind spot on the left side of the target vehicle; or, the steering information of the target vehicle indicates that the target vehicle is turning right, And the position of the target object indicates that when the target object is in the blind spot on the right side of the target vehicle, it is considered that the target vehicle has a greater impact on the safety of the target object while driving. For example, the target vehicle may collide with pedestrians while driving, and the monitoring results can include the highest level monitoring results. Here, for example, the monitoring results can also be divided into multiple levels, such as level 1, level 2, level 3, and level 4; the higher the level, the greater the impact on the driving safety of the target vehicle, and the corresponding warning information also The corresponding representation has a great influence on the driving safety of the target vehicle.

Taking the first monitoring result as an example, the first monitoring result, for example, corresponds to level one, and includes a relatively high-frequency "beep" audio, or a voice prompt message "the vehicle is too close, please drive carefully".

In addition, the first monitoring result may be further refined according to the location of the target object. Taking the driving state of the target device as driving to the left, and the position of the target object indicates that there is a target object on the left side of the target vehicle, if the first monitoring result indicates that the target device is constantly approaching the target object, then gradually increase ", or generate warning messages with more accurate prompt information such as "the current distance is 1 meter from the pedestrian on the left", "the current distance is 0.5 meters from the pedestrian on the left".

In another possible implementation, the steering information of the target vehicle indicates that the target vehicle is turning left, and the position of the target object indicates that the target object is in the blind spot on the right side of the target vehicle; or, the steering information of the target vehicle indicates that the target vehicle is turning right , and the position of the target object indicates that when the target object is in the blind spot on the left side of the target vehicle, it is considered that the target vehicle has a certain probability of causing a certain impact on safety when driving. For example, when a pedestrian approaches the target vehicle, a collision may occur, then monitor The results can correspond to the second level, including the "beep" sound audio with a lower frequency than the monitoring results corresponding to the first level, or the voice prompt message "currently close to pedestrians, please drive carefully".

In addition, the type information of the target object may also include, for example, a vehicle; here, the vehicle is other vehicles except the target vehicle.

Similar to the above method of determining the monitoring result when the type information indicates that the target object is a pedestrian, in a possible implementation, the steering information of the target vehicle indicates that the target vehicle is turning left, and the current position of the target object indicates that the target object is in The blind spot on the left side of the target vehicle; or, when the steering information of the target vehicle indicates that the target vehicle is turning right, and the current position of the target object indicates that the target object is in the blind spot on the right side of the target vehicle, it is considered that the target vehicle has a greater impact on safety when driving, For example, the target vehicle may collide with other vehicles when turning, and the monitoring results may include monitoring results corresponding to three levels.

In addition, the monitoring results can be further refined according to the driving status. Take the driving state of the target vehicle as an example of the target vehicle turning left, and the monitoring result indicates that there is a target object on the left side of the target vehicle. Frequency, or generate warning information with more accurate prompt information such as "the current distance is 1 meter from the left vehicle" and "the current distance is 0.5 meters from the left vehicle".

In another possible implementation, the steering information of the target vehicle indicates that the target vehicle is turning left, and the position of the target object indicates that the target object is in the blind spot on the right side of the target vehicle; or, the steering information of the target vehicle indicates that the target vehicle is turning right , and the position of the target object indicates that when the target object is in the blind spot on the left side of the target vehicle, it is considered that the target object has a certain probability of affecting the safety of the target vehicle when driving. For example, when the target vehicle is driving, other vehicles may collide with the target vehicle When a vehicle collides, the monitoring result can correspond to level 4, including a "beep" audio at a lower frequency than the monitoring result corresponding to level 3, or a voice prompt message "the vehicle is currently close, please drive carefully".

In this way, by generating the warning information more accurately and quickly, the driver controlling the target vehicle can be guided to drive more safely.

In addition, the monitoring results may also include vehicle control instructions, for example. Correspondingly, the driving state of the target vehicle may include steering information of the target vehicle, for example.

Here, since the type information and position of the target object can be obtained more efficiently and accurately, monitoring results including vehicle control commands can also be generated to control the safe driving of the driving device and the like. Wherein, the driving device is, for example but not limited to, any of the following: an autonomous vehicle, a vehicle equipped with an Advanced Driving Assistance System (Advanced Driving Assistance System, ADAS), or a robot.

In a specific implementation, when generating monitoring results according to the type information and location of the target object and the driving state of the target vehicle, for example, a vehicle control instruction may be generated according to the type information and location of the target object and the steering information of the target vehicle.

Here, according to the type information and position of the target object and the steering information of the target vehicle, when generating the vehicle control command, for example, the target vehicle can be determined according to the type information and position of the target object to determine the generated vehicle control command, so as to ensure that the target vehicle can avoid Collide with the target object to ensure safe driving.

In this way, the monitoring results are more conducive to deployment in the intelligent driving device, improving the safety of the intelligent driving device during the automatic driving control process, that is, it can better meet the needs of the automatic driving field.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

Based on the same technical idea, the embodiment of the present disclosure also provides a blind spot monitoring device corresponding to the blind spot monitoring method. Since the problem-solving principle of the device in the embodiment of the present disclosure is similar to the blind spot monitoring method described above in the embodiment of the present disclosure, the implementation of the device Reference can be made to the implementation of the method, and repeated descriptions will not be repeated.

Referring to FIG. 12 , it is a schematic diagram of a blind spot monitoring device provided by an embodiment of the present disclosure. The blind spot monitoring device includes: an acquisition module 121, a detection module 122, a determination module 123, and a generation module 124; wherein,

An acquisition module 121, configured to acquire the current frame monitoring image collected by the acquisition device on the target vehicle;

The detection module 122 is configured to perform object detection on the current frame monitoring image to obtain type information and positions of objects included in the image;

A determining module 123, configured to determine the target object located in the blind spot of the target vehicle according to the position of the object and the blind spot of the target vehicle;

The generation module 124 is configured to generate a monitoring result according to the type information and location of the target object and the driving state of the target vehicle.

In a possible implementation manner, the monitoring result includes warning information, and the driving state of the target vehicle includes steering information of the target vehicle; The driving state of the target vehicle, when generating the monitoring result, is used to: determine the level of warning information according to the type information and position of the target object and the steering information of the target vehicle; generate a certain level of warning information and prompt .

In a possible implementation manner, the monitoring results include vehicle control instructions, and the driving state of the target vehicle includes steering information of the target vehicle; And the driving state of the target vehicle, when generating the monitoring result, it is used to: generate the vehicle control instruction according to the type information and position of the target object and the steering information of the target vehicle; the blind spot monitoring device also includes The control module 125 is configured to: control the running of the target vehicle based on the vehicle control instruction.

In a possible implementation manner, when determining the target object located in the blind spot of the target vehicle according to the position of the object and the blind spot of the target vehicle, the determining module 123 is configured to: The position of the object in the current frame monitoring image is determined to determine the current first distance information between the target vehicle and the object in the current frame monitoring image; according to the current first distance information, it is determined that the target vehicle is located Target objects in the blind zone of the field of view.

In a possible implementation manner, the determining module 123 determines the current first distance between the target vehicle and the object in the current frame monitoring image according to the position of the object in the current frame monitoring image information, used for: determining the distance to be adjusted between the target vehicle and the object in the current frame monitoring image based on the current frame monitoring image; based on the multiple frames of the object collected by the acquisition device Scale change information between scales in two adjacent frames of monitoring images in the historical frame monitoring images, and the distance between the object and the target vehicle in each frame of the historical frame monitoring images in the multiple frames of historical frame monitoring images The historical first distance information is adjusted to the to-be-adjusted distance information to obtain the current first distance information between the target vehicle and the object.

In a possible implementation manner, when the determining module 123 adjusts the distance information to be adjusted to obtain the current first distance information between the target vehicle and the object, it is configured to: The distance information to be adjusted is adjusted until the error amount of the scale change information is the smallest, and the adjusted distance information is obtained; wherein the error amount is based on the distance information to be adjusted, the scale change information, and the multi-frame history Determining the historical first distance information corresponding to each frame of the historical frame monitoring image in the frame monitoring image; determining the current first distance information based on the adjusted distance information.

In a possible implementation manner, before determining the current first distance information based on the adjusted distance information, the determining module 123 is further configured to: monitor the distance of the object in the image based on the current frame position, and the calibration parameters of the acquisition device to determine the current second distance information; when the determination module 123 determines the current first distance information based on the adjusted distance information, it is configured to: based on the current The second distance information, the historical second distance information between the object and the target vehicle in each frame of the historical frame monitoring image of the multiple frames of historical frame monitoring images, the history corresponding to the frame of historical frame monitoring images First distance information and the adjusted distance information, determining distance offset information for the adjusted distance information; adjusting the adjusted distance information based on the distance offset information to obtain the current First distance information.

In a possible implementation manner, the determination module 123 is based on the current second distance information, and the distance between the object and the target vehicle in each frame of the multiple frames of historical frame monitoring images. The historical second distance information between the frames, the historical first distance information corresponding to the historical frame monitoring image, and the adjusted distance information, when determining the distance offset information for the adjusted distance information, used : Based on the current second distance information and the historical second distance information corresponding to each frame of the multiple frames of historical frame monitoring images, determine each frame in the multiple frames of historical frame monitoring images The historical second distance information corresponding to the historical frame monitoring image and the first linear fitting coefficient of the first fitting curve fitted by the current second distance information; based on each of the multi-frame historical frame monitoring images The historical first distance information corresponding to the frame historical frame monitoring image and the adjusted distance information, determine the historical first distance information corresponding to each frame of the historical frame monitoring image in the multi-frame historical frame monitoring image The second linear fitting coefficient of the second fitting curve fitted with the adjusted distance information; based on the first linear fitting coefficient and the second linear fitting coefficient, determine the adjusted The distance offset information of the distance information.

In a possible implementation manner, when determining the current second distance information based on the position of the object in the current frame monitoring image and the calibration parameters of the acquisition device, the determining module 123 is used to : Based on the position information of the detection frame of the object in the current frame monitoring image, obtain the pixel coordinate value of the set corner point in the detection frame; based on the pixel coordinate value of the set corner point, the acquisition The calibration parameters of the device and the pixel coordinate values of the vanishing point of the lane line used when determining the calibration parameters of the acquisition device determine the current second distance information.

In a possible implementation manner, the calibration parameters of the collection device include a first height value of the collection device relative to the ground and a focal length of the collection device; The pixel coordinate value of the pixel coordinate value, the calibration parameter of the collection device, and the pixel coordinate value of the vanishing point of the lane line used when determining the calibration parameter of the collection device, when determining the current second distance information, it is used to: based on the The pixel coordinate value of the disappearing point of the lane line and the pixel coordinate value of the set corner point in the detection frame determine the first pixel height value of the collection device relative to the ground; based on the pixel coordinate value of the set corner point, Determining a second pixel height value of the object in the current frame monitoring image relative to the ground; based on the first pixel height value, the second pixel height value, and the first height value, determine the object A second height value relative to the ground; determining the current second distance information based on the second height value, the focal length of the acquisition device, and the second pixel height value.

In a possible implementation manner, the determining module 123 is configured to: when determining the distance information to be adjusted between the target vehicle and the object in the current frame monitoring image based on the current frame monitoring image: Obtaining scale change information between the scale of the object in the current frame monitoring image and the scale in the historical frame monitoring image adjacent to the current frame monitoring image; based on the scale change information and the The historical first distance information corresponding to the historical frame monitoring image adjacent to the current frame monitoring image is used to determine the to-be-adjusted distance information.

In a possible implementation manner, the determination module 123 determines the scale change information of the object between the scales of two adjacent frames of monitoring images in the following manner: respectively extracting a plurality of feature points contained in the object in The first position information in the previous frame of the monitoring image in the two adjacent frames of monitoring images, and the second position information in the following frame of monitoring images; based on the first position information and the second position information, Determine scale change information of the object between scales in two adjacent frames of monitoring images.

In a possible implementation manner, the determination module 123 determines the scale change information between the scales of the object in two adjacent frames of monitoring images based on the first position information and the second position information. , for: based on the first position information, determine the first scale value of the target line segment formed by the plurality of feature points contained in the object in the previous frame monitoring image; based on the second position information, Determining a second scale value of the target line segment in the next frame of the monitoring image; based on the first scale value and the second scale value, determining the difference between the scales of the object in two adjacent frames of monitoring images Information about scale changes between them.

Corresponding to the blind spot monitoring method in FIG. 1, an embodiment of the present disclosure also provides an electronic device 1300. As shown in FIG. 13, it is a schematic structural diagram of the electronic device 1300 provided by the embodiment of the present disclosure, including:

Processor 10, memory 20, and bus 30; memory 20 is used for storing and executing instructions, and includes memory 210 and external memory 220; memory 210 here is also called internal memory, and is used for temporarily storing computing data in processor 10, and The data exchanged by the external memory 220 such as hard disk, the processor 10 exchanges data with the external memory 220 through the memory 210, when the electronic device 1300 is running, the processor 10 communicates with the memory 20 through the bus 30, so that the processor 10 executes the following instructions : Obtain the current frame monitoring image collected by the acquisition device on the target vehicle; perform object detection on the current frame monitoring image to obtain the type information and position of the object included in the current frame monitoring image; according to the position of the object and the blind spot of the target vehicle, determining a target object located in the blind spot of the target vehicle; generating a monitoring result according to the type information and position of the target object and the driving state of the target vehicle.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the blind spot monitoring method described in the foregoing method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

The embodiment of the present disclosure also provides a computer program product, the computer program product carries a program code, and the instructions included in the program code can be used to execute the steps of the blind spot monitoring method described in the above method embodiment, for details, please refer to the above method The embodiment will not be repeated here.

Wherein, the above-mentioned computer program product may be specifically implemented by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. wait.

Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and other media that can store program codes.

Finally, it should be noted that: the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, rather than limit them, and the protection scope of the present disclosure is not limited thereto, although referring to the aforementioned The embodiments have described the present disclosure in detail, and those skilled in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present disclosure Changes can be easily imagined, or equivalent replacements can be made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in this disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be defined by the protection scope of the claims.

Claims (12)

1. The blind area monitoring method is characterized by comprising the following steps of:

acquiring a current frame monitoring image acquired by acquisition equipment on a target vehicle;

performing object detection on the current frame monitoring image to obtain type information and position of an object included in the current frame monitoring image;

Determining a target object positioned in a visual field blind area of the target vehicle according to the position of the object and the visual field blind area of the target vehicle;

generating a monitoring result according to the type information and the position of the target object and the driving state of the target vehicle;

wherein the determining the target object located in the blind field of view of the target vehicle according to the position of the object and the blind field of view of the target vehicle comprises: acquiring scale change information between a scale of the object in the current frame monitoring image and a scale in a historical frame monitoring image adjacent to the current frame monitoring image; determining distance information to be adjusted based on the scale change information and historical first distance information corresponding to a historical frame monitoring image adjacent to the current frame monitoring image; adjusting the distance information to be adjusted until the error amount of the scale change information is minimum, and obtaining adjusted distance information; the error amount is determined based on the distance information to be adjusted, the scale change information and historical first distance information corresponding to each frame of historical frame monitoring image in the multi-frame historical frame monitoring images; determining current first distance information based on the adjusted distance information; and determining the target object positioned in the blind area of the visual field of the target vehicle according to the current first distance information.

2. The blind area monitoring method according to claim 1, wherein the monitoring result includes warning information, and the driving state of the target vehicle includes steering information of the target vehicle;

and generating a monitoring result according to the type information and the position of the target object and the driving state of the target vehicle, wherein the monitoring result comprises the following steps:

determining the level of alarm information according to the type information and the position of the target object and the steering information of the target vehicle;

and generating and prompting the alarm information of the determined level.

3. The blind area monitoring method according to claim 1, wherein the monitoring result includes a vehicle control instruction, and the driving state of the target vehicle includes steering information of the target vehicle;

and generating a monitoring result according to the type information and the position of the target object and the driving state of the target vehicle, wherein the monitoring result comprises the following steps:

generating the vehicle control instruction according to the type information and the position of the target object and the steering information of the target vehicle;

the blind area monitoring method further comprises the following steps: and controlling the target vehicle to run based on the vehicle control instruction.

4. The blind zone monitoring method according to claim 1, characterized in that the blind zone monitoring method further comprises, before determining the current first distance information based on the adjusted distance information:

determining current second distance information based on the position of the object in the current frame monitoring image and calibration parameters of the acquisition equipment;

the determining the current first distance information based on the adjusted distance information includes:

determining distance offset information for the adjusted distance information based on the current second distance information, historical second distance information between the object and the target vehicle in each frame of historical frame monitoring image in the multi-frame historical frame monitoring image, the historical first distance information corresponding to the frame of historical frame monitoring image, and the adjusted distance information;

and adjusting the adjusted distance information based on the distance offset information to obtain the current first distance information.

5. The blind zone monitoring method according to claim 4, wherein the determining distance offset information for the adjusted distance information based on the current second distance information, historical second distance information between the object and the target vehicle in each of the plurality of frames of historical frame monitoring images, the historical first distance information corresponding to the frame of historical frame monitoring image, and the adjusted distance information includes:

Determining a first linear fitting coefficient of a first fitting curve which is formed by fitting the historical second distance information corresponding to each frame of history frame monitoring image in the multi-frame history frame monitoring image and the current second distance information based on the current second distance information and the historical second distance information corresponding to each frame of history frame monitoring image in the multi-frame history frame monitoring image;

determining a second linear fitting coefficient of a second fitting curve which is formed by fitting the historical first distance information corresponding to each frame of history frame monitoring image in the multi-frame history frame monitoring image and the adjusted distance information based on the historical first distance information corresponding to each frame of history frame monitoring image in the multi-frame history frame monitoring image and the adjusted distance information;

distance offset information for the adjusted distance information is determined based on the first linear fit coefficient and the second linear fit coefficient.

6. The blind zone monitoring method according to claim 4 or 5, characterized in that the determining the current second distance information based on the position of the object in the current frame monitoring image and the calibration parameters of the acquisition device includes:

Acquiring pixel coordinate values of set corner points in a detection frame based on the position information of the object in the current frame monitoring image;

and determining the current second distance information based on the pixel coordinate values of the set corner points, the calibration parameters of the acquisition equipment and the pixel coordinate values of the lane line vanishing points used in determining the calibration parameters of the acquisition equipment.

7. The blind zone monitoring method of claim 6 wherein the calibration parameters of the acquisition device include a first height value of the acquisition device relative to ground and a focal length of the acquisition device;

the determining the current second distance information based on the pixel coordinate values of the set corner points, the calibration parameters of the collecting device, and the pixel coordinate values of the lane line vanishing points used in determining the calibration parameters of the collecting device includes:

determining a first pixel height value of the acquisition equipment relative to the ground based on the pixel coordinate value of the lane line vanishing point and the pixel coordinate value of the set corner point in the detection frame;

determining a second pixel height value of the object in the current frame monitoring image relative to the ground based on the pixel coordinate values of the set corner points;

Determining a second height value of the object relative to the ground based on the first pixel height value, the second pixel height value, and the first height value;

the current second distance information is determined based on the second height value, the focal length of the acquisition device, and the second pixel height value.

8. The blind area monitoring method according to claim 1, characterized in that the scale change information of the object between scales in two adjacent frames of monitoring images is determined in the following manner:

respectively extracting first position information of a plurality of feature points contained in the object in a previous frame of monitoring image and second position information in a next frame of monitoring image in the two adjacent frames of monitoring images;

and determining scale change information of the object between scales in two adjacent frames of monitoring images based on the first position information and the second position information.

9. The blind zone monitoring method according to claim 8, wherein the determining scale change information of the object between scales in two adjacent frames of monitoring images based on the first position information and the second position information includes:

Determining a first scale value of a target line segment formed by a plurality of feature points contained in the object in the previous frame of monitoring image based on the first position information;

determining a second scale value of the target line segment in the monitoring image of the later frame based on the second position information;

and determining scale change information of the object between scales in two adjacent frames of monitoring images based on the first scale value and the second scale value.

10. A blind zone monitoring device, comprising:

the acquisition module is used for acquiring a current frame monitoring image acquired by acquisition equipment on the target vehicle;

the detection module is used for carrying out object detection on the current frame monitoring image to obtain type information and position of an object included in the image;

the determining module is used for determining a target object positioned in the visual field blind area of the target vehicle according to the position of the object and the visual field blind area of the target vehicle;

the generation module is used for generating a monitoring result according to the type information and the position of the target object and the driving state of the target vehicle;

the determining module is specifically configured to, when determining a target object located in a blind area of a field of view of the target vehicle according to a position of the object and the blind area of the field of view of the target vehicle: acquiring scale change information between a scale of the object in the current frame monitoring image and a scale in a historical frame monitoring image adjacent to the current frame monitoring image; determining distance information to be adjusted based on the scale change information and historical first distance information corresponding to a historical frame monitoring image adjacent to the current frame monitoring image; adjusting the distance information to be adjusted until the error amount of the scale change information is minimum, and obtaining adjusted distance information; the error amount is determined based on the distance information to be adjusted, the scale change information and historical first distance information corresponding to each frame of historical frame monitoring image in the multi-frame historical frame monitoring images; determining current first distance information based on the adjusted distance information; and determining the target object positioned in the blind area of the visual field of the target vehicle according to the current first distance information.

11. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication over the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the blind zone monitoring method of any one of claims 1 to 9.

12. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, performs the steps of the blind zone monitoring method according to any one of claims 1 to 9.