CN112784639A - Intersection detection, neural network training and intelligent driving method, device and equipment - Google Patents

Abstract

This embodiment discloses a method, device, electronic device and computer storage medium for intersection detection, neural network training and intelligent driving. The intersection detection method includes: extracting features from a road image to obtain a feature map of the road image; The feature map of the road image determines the detection frame of the intersection on the road shown by the road image; the detection frame of the intersection represents the area of the intersection in the road image, and the lower border of the detection frame of the intersection On the road surface of the road; according to the lower border of the detection frame of the intersection, determine the distance between the device that collects the road image and the intersection. In this way, even if a clear traffic light or ground stop line image cannot be obtained, or there is no traffic light or ground stop line at the intersection, the embodiment of the present disclosure can also implement intersection detection according to the feature map of the road image, so as to determine whether to collect the road. Image the distance between the device and the intersection.

Description

Intersection detection, neural network training and intelligent driving method, device and equipment

technical field

The present disclosure relates to computer vision processing technology, and in particular, to a method, device, electronic device, and computer storage medium for intersection detection, neural network training, and intelligent driving.

Background technique

In recent years, with the improvement of living standards and the improvement of assisted driving technology, more and more needs related to assisted driving have been put forward, and more and more scholars and companies have applied deep learning to assisted driving solutions. When performing assisted driving or autonomous driving tasks, intersection detection and determining the distance between the vehicle and the intersection according to the detected intersection are very important tasks.

SUMMARY OF THE INVENTION

The embodiments of the present disclosure are expected to provide a technical solution for intersection detection.

An embodiment of the present disclosure provides a method for detecting an intersection, and the method includes:

Perform feature extraction on the road image to obtain a feature map of the road image;

According to the feature map of the road image, the detection frame of the intersection on the road shown by the road image is determined; the detection frame of the intersection represents the area of the intersection in the road image, and the lower part of the detection frame of the intersection is the border is on the pavement of said road;

According to the lower border of the detection frame of the intersection, the distance between the device collecting the road image and the intersection is determined.

Optionally, the method further includes:

According to the feature map of the road image, it is determined that there is no intersection on the road shown by the road image.

Optionally, determining the distance between the device that collects the road image and the intersection according to the lower border of the detection frame of the intersection includes:

According to the position of the lower border of the detection frame of the intersection in the road image and the coordinate conversion relationship between the plane of the road image and the road surface of the road, it is determined that the lower border of the detection frame of the intersection is at the location of the intersection. the position on the said road;

According to the position of the lower border of the detection frame of the intersection on the road and the position of the device that collects the road image on the road, the distance between the device that collects the road image and the intersection is obtained .

Optionally, the method is performed by a neural network, and the neural network is obtained by training a sample image and a labeling result of the sample image, and the labeling result of the sample image includes the labeling frame of the intersection on the road shown in the positive sample image, The labeling frame represents the position of the intersection in the positive sample image, and the lower border of the labeling frame is on the road surface of the road shown in the positive sample image.

Embodiments of the present disclosure also provide a neural network training method, including:

Perform feature extraction on the sample image to obtain a feature map of the sample image;

determining the detection result of the sample image according to the feature map of the sample image;

Adjust the network parameter value of the neural network according to the labeling result of the sample image and the detection result;

When the sample image is a positive sample image, the labeling result of the sample image is the labeling frame of the intersection on the road shown by the positive sample image, and the labeling frame represents the position of the intersection in the positive sample image , and the lower border of the labeling frame of the intersection on the road shown in the positive sample image is on the road surface of the road shown in the positive sample image.

Optionally, the positive sample image includes a stop line of a road intersection, and the lower border of the label box of the intersection on the road shown in the positive sample image is aligned with the stop line.

Optionally, the difference between the heights of the annotation boxes in the multiple positive sample images including the same intersection is within a preset range.

Optionally, when the sample image is a negative sample image, there is no intersection on the road in the negative sample image, and the annotation result of the sample image includes that there is no annotation frame in the negative sample image.

The embodiment of the present disclosure also provides an intelligent driving method, including:

Get road images;

According to any one of the above-mentioned intersection detection methods, the intersection detection is performed on the road image;

The driving control of the device is performed according to the distance between the intelligent driving device that collects the road image and the intersection.

An embodiment of the present disclosure also provides an intersection detection device, the device includes a first extraction module, a detection module, and a first determination module; wherein,

a first extraction module, configured to perform feature extraction on a road image to obtain a feature map of the road image;

a detection module, configured to determine the detection frame of the intersection on the road shown in the road image according to the feature map of the road image; the detection frame of the intersection represents the area of the intersection in the road image, and the intersection The lower border of the detection frame is on the road surface of the road;

The first determination module is configured to determine the distance between the device collecting the road image and the intersection according to the lower border of the detection frame of the intersection.

Optionally, the detection module is further configured to determine, according to the feature map of the road image, that there is no intersection on the road shown by the road image.

Optionally, the first determination module is configured to, according to the position of the lower border of the detection frame of the intersection in the road image and the coordinate conversion relationship between the plane of the road image and the road surface of the road , determine the position of the lower border of the detection frame of the intersection on the road; according to the position of the lower border of the detection frame of the intersection on the road and the position of the device collecting the road image on the road position to obtain the distance between the device that captures the road image and the intersection.

Optionally, the device is implemented based on a neural network, and the neural network is obtained by training a sample image and an annotation result of the sample image, and the annotation result of the sample image includes the annotation of the intersection on the road shown in the positive sample image. frame, the labeling frame represents the position of the intersection in the positive sample image, and the lower border of the labeling frame is on the road surface of the road shown in the positive sample image.

The embodiment of the present disclosure also provides a neural network training device, the device includes: a second extraction module, a second determination module and an adjustment module, wherein,

a second extraction module, configured to perform feature extraction on the sample image to obtain a feature map of the sample image;

a second determination module, configured to determine the detection result of the sample image according to the feature map of the sample image;

an adjustment module, configured to adjust the network parameter value of the neural network according to the labeling result of the sample image and the detection result;

When the sample image is a positive sample image, the labeling result of the sample image is the labeling frame of the intersection on the road shown by the positive sample image, and the labeling frame represents the position of the intersection in the positive sample image , and the lower border of the labeling frame of the intersection on the road shown in the positive sample image is on the road surface of the road shown in the positive sample image.

Optionally, the positive sample image includes a stop line of a road intersection, and the lower border of the label box of the intersection on the road shown in the positive sample image is aligned with the stop line.

Optionally, the difference between the heights of the annotation boxes in the multiple positive sample images including the same intersection is within a preset range.

Optionally, when the sample image is a negative sample image, there is no intersection on the road in the negative sample image, and the annotation result of the sample image includes that there is no annotation frame in the negative sample image.

An embodiment of the present disclosure further provides an intelligent driving device, the device includes: an acquisition module and a processing module, wherein the acquisition module is used to acquire road images;

The processing module is configured to perform intersection detection on the road image according to any one of the above-mentioned intersection detection methods; and control the driving of the device according to the distance between the intelligent driving device that collects the road image and the intersection.

Embodiments of the present disclosure also provide an electronic device, including a processor and a memory for storing a computer program that can be executed on the processor; wherein,

The processor is configured to run the computer program to execute any of the above-mentioned intersection detection methods or any of the above-mentioned neural network training methods or any of the above-mentioned intelligent driving testing methods.

Embodiments of the present disclosure also provide a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, implements any of the above-mentioned intersection detection methods or any of the above-mentioned neural network training methods or any of the above-mentioned methods Intelligent driving test method.

In the method, device, electronic device, and computer storage medium for intersection detection, neural network training, and intelligent driving proposed in the embodiments of the present disclosure, the intersection detection method includes: performing feature extraction on a road image to obtain a feature map of the road image; The feature map of the road image is used to determine the detection frame of the intersection on the road shown by the road image; the detection frame of the intersection represents the area of the intersection in the road image; The lower border of the detection frame is on the road surface of the road, so the distance between the device collecting the road image and the intersection can be determined according to the lower border of the detection frame of the intersection; The ground stop line image, or when there are no traffic lights or ground stop lines at the intersection, the embodiment of the present disclosure can also implement intersection detection according to the feature map of the road image, so as to determine the distance between the device that collects the road image and the intersection .

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

FIG. 1 is a flowchart of an intersection detection method according to an embodiment of the present disclosure;

2 is a flowchart of a neural network training method according to an embodiment of the present disclosure;

FIG. 3 is an example diagram of an embodiment of the present disclosure using a trained neural network for intersection detection;

4 is a flowchart of an intelligent driving method according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of the composition and structure of an intersection detection device according to an embodiment of the present disclosure;

6 is a schematic diagram of the composition and structure of a neural network training apparatus according to an embodiment of the present disclosure;

7 is a schematic diagram of the composition and structure of an intelligent driving device according to an embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.

Detailed ways

The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are only used to explain the present disclosure, but not to limit the present disclosure. In addition, the embodiments provided below are only some of the embodiments for implementing the present disclosure, rather than all the embodiments for implementing the present disclosure. In the case of no conflict, the technical solutions described in the embodiments of the present disclosure can be combined in any way. implement.

It should be noted that, in the embodiments of the present disclosure, the terms "comprising", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements not only includes the explicitly stated elements, but also other elements not expressly listed or inherent to the implementation of the method or apparatus. Without further limitation, an element defined by the phrase "comprises a..." does not preclude the presence of additional related elements (eg, steps in a method or a device) in which the element is included. A unit in an apparatus, for example, a unit may be part of a circuit, part of a processor, part of a program or software, etc.).

For example, the intersection detection, neural network training method, and intelligent driving method provided by the embodiments of the present disclosure include a series of steps, but the intersection detection, neural network training method, and intelligent driving method provided by the embodiments of the present disclosure are not limited to the described steps , Similarly, the intersection detection device, neural network training device, and intelligent driving device provided by the embodiments of the present disclosure include a series of modules, but the devices provided by the embodiments of the present disclosure are not limited to including the explicitly recorded modules, and may also include modules for obtaining Related information, or modules that need to be set when processing based on information.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

Embodiments of the present disclosure can be applied to computer systems consisting of terminals and servers, and can operate with numerous other general-purpose or special-purpose computing system environments or configurations. Here, the terminals may be thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, etc., and the server may be a server computer Systems Small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above, etc.

Electronic devices such as terminals, servers, etc., may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer systems/servers may be implemented in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

In assisted driving or automatic driving tasks, it is necessary to perceive surrounding information through cameras and radar, and at the same time, it is necessary to give accurate decision-making information, such as acceleration, avoidance, deceleration, etc.; the structure of the intersection area is often complex, and the vehicle is far away from the intersection. It is particularly important how to accurately predict the intersection and estimate the distance. Generally speaking, the purpose of accurate detection of intersection areas is to effectively provide sufficient reaction time for automatic driving decisions, and to reserve sufficient time for vehicle deceleration. In related technologies, information such as traffic lights at intersections or ground stop lines captured by the camera of the vehicle is usually used for judgment; when the vehicle is far away from the intersection, clear images of traffic lights or ground stop lines cannot be obtained, resulting in the above intersection detection. The scheme cannot perform accurate intersection detection; in addition, some intersections do not have traffic lights or ground stop lines, which will cause the above intersection detection scheme to fail to achieve intersection detection.

In view of the above-mentioned problems, in some embodiments of the present disclosure, an intersection detection method is proposed, and the embodiments of the present disclosure can be applied to scenarios such as automatic driving and assisted driving.

FIG. 1 is a flowchart of an intersection detection method according to an embodiment of the present disclosure. As shown in FIG. 1 , the flowchart may include:

Step 101: Perform feature extraction on the road image to obtain a feature map of the road image.

Here, the road image is an image that needs to be detected at the intersection. Exemplarily, the format of the road image may be Joint Photographic Experts Group (Joint Photographic Experts GROUP, JPEG), Bitmap (Bitmap, BMP), Portable Network Graphics (Portable Network Graphics, PNG) or other formats; it should be noted that, The format of the road image is only exemplified here, and the embodiment of the present disclosure does not limit the format of the sample image.

In practical applications, road images can be acquired from a local storage area or network, or road images can be acquired by using an image acquisition device. Here, the image acquisition device may include a camera installed on the vehicle, etc.; One or more cameras are set up to capture images of the road ahead of the vehicle.

In the embodiment of the present disclosure, the feature map of the road image may be used to represent at least one of the following features of the road image: color feature, texture feature, shape feature, and spatial relationship feature. For the implementation of this step, in an example, a scale-invariant feature transform (SIFT) method or a Histogram of Oriented Gradient (HOG) feature extraction method can be used to extract the features of the road image Figure; in another example, a pre-trained neural network for extracting image feature maps can also be used to perform feature extraction on road images.

Step 102: Determine the detection frame of the intersection on the road shown in the road image according to the feature map of the road image; the detection frame of the intersection represents the area of the intersection in the road image, and the lower border of the detection frame of the intersection is at the intersection. on the road surface.

In the embodiment of the present disclosure, it can be determined whether the road shown in the road image has an intersection according to the feature map of the road image, and the judgment result can be obtained. Obviously, the judgment result includes the following two situations: the road shown in the road image has an intersection, and the road shown in the road image has an intersection. There is no intersection on the road shown in the road image; when there is an intersection on the road shown in the road image, the detection frame of the intersection on the road shown in the road image can be determined according to the feature map of the road image, and the detection frame of the intersection is output; When there is no intersection on the road shown, nothing is output.

In practical applications, when there is an intersection on the road shown in the road image, a pre-trained neural network for extracting intersection detection frames can be used to determine the detection frame of the intersection on the road shown in the road image.

In the embodiment of the present disclosure, the shape of the detection frame of the intersection is not limited. For example, the shape of the detection frame of the intersection may be a rectangle, a trapezoid, etc.; After the feature map of the image is input to the neural network for extracting the intersection detection frame, the neural network for extracting the intersection detection frame can output the rectangular intersection detection frame; in another specific example, the road shown in the road image is not. There is an intersection, and after the feature map of the road image is input to the neural network for extracting intersection detection frame, the neural network for extracting intersection detection frame does not output any data.

Step 103: According to the lower border of the detection frame of the intersection, determine the distance between the device that collects the road image and the intersection.

It can be understood that, since the lower border of the detection frame of the intersection is on the road surface, the position of the intersection can be determined according to the lower border of the detection frame of the intersection, and further, combined with the known position of the device that collects road images, it is possible to determine the position of the intersection. Determine the distance between the device capturing the road image and the intersection.

In practical applications, steps 101 to 103 can all be implemented by a processor in an electronic device, and the above-mentioned processor can be an Application Specific Integrated Circuit (ASIC) or a Digital Signal Processor (DSP) , Digital Signal Processing Device (DSPD), Programmable Logic Device (Programmable Logic Device, PLD), Field Programmable Gate Array (Field-Programmable Gate Array, FPGA), Central Processing Unit (Central Processing Unit, CPU), At least one of a controller, a microcontroller, and a microprocessor.

It can be seen that, in the embodiment of the present disclosure, first, a feature map of a road image is performed to obtain a feature map of the road image; then, according to the feature map of the road image, the In addition, since the lower border of the detection frame of the intersection in the embodiment of the present disclosure is on the road surface of the road, it can be determined according to the lower border of the detection frame of the intersection, the difference between the device that collects the road image and the intersection can be determined. In this way, even if a clear traffic light or ground stop line image cannot be obtained, or there is no traffic light or ground stop line at the intersection, the embodiment of the present disclosure can implement intersection detection according to the feature map of the road image, so as to determine The distance between the device capturing the road image and the intersection.

In addition, the intersection detection method of the embodiment of the present disclosure has strong universality. When at least one camera is installed on the vehicle, the intersection detection can be accurately performed on the image in front of the vehicle, which can be realized when the distance from the intersection is far away. Intersection detection is conducive to providing sufficient reaction time for driving decisions and ensuring driving safety. For example, it can provide sufficient reaction time for braking.

Optionally, for some road images that do not contain intersections, it can be confirmed that there are no intersections on the road shown in the road images directly according to the feature map of the road images, which is helpful for providing assistance in driving decision-making and ensuring driving safety.

For the implementation of determining the distance between the device that collects the road image and the intersection according to the lower border of the detection frame of the intersection, for example, the position of the lower border of the detection frame of the intersection in the road image and the plane of the road image can be used. According to the coordinate conversion relationship between the road surface and the road surface, determine the position of the lower border of the intersection detection frame on the road; according to the position of the lower border of the intersection detection frame on the road and the position of the device that collects the road image on the road, Find the distance between the device that captures the road image and the intersection.

In the relevant intersection detection scheme, the distance between the intersection and the vehicle cannot be determined; while in the embodiment of the present disclosure, when the device for collecting road images is located in the vehicle, the distance between the device for collecting road images and the intersection can be determined. The distance is the distance between the vehicle and the intersection, that is to say, in this embodiment of the present disclosure, it can be considered that the lower border of the intersection detection frame will fit with the road surface, and according to the position of the lower border of the intersection detection frame in the road image, it can be accurately determined. Estimating the distance between the vehicle and the intersection is conducive to providing sufficient reaction time for driving decisions and ensuring driving safety.

In one embodiment, according to the coordinate conversion relationship between the plane of the road image and the road surface of the road, the position coordinates of the lower border of the detection frame of the intersection can be converted to the world coordinate system, and the lower border of the detection frame of the intersection can be obtained in the world. The position of the coordinate system, that is, the position of the lower border of the detection frame of the intersection on the road.

In practical applications, the plane of the road image and the pavement of the road are two different planes. In this way, a homography (Homography) matrix can be used to represent the coordinate transformation relationship between the plane of the road image and the pavement of the road. According to the homography matrix, the position coordinates of the lower border of the detection frame of the intersection are converted to the world coordinate system; the homography matrix can be calculated from the road image and some corresponding points in the world coordinate system. Based on this homography matrix , the position of each point in the road image in the world coordinate system can be accurately obtained.

As an embodiment, the above-mentioned intersection detection method can be performed by a neural network, and the above-mentioned neural network is obtained by training a sample image and the labeling result of the sample image, and the labeling result of the sample image includes the labeling frame of the intersection on the road shown by the positive sample image. , the labeling frame of the intersection on the road shown in the positive sample image represents the position of the intersection in the positive sample image, and the lower border of the labeling frame of the intersection on the road shown in the positive sample image is on the side of the road shown in the positive sample image. on the road.

Here, the format of the sample image can be Joint Photographic Experts Group (Joint Photographic ExpertsGROUP, JPEG), Bitmap (Bitmap, BMP), Portable Network Graphics (Portable Network Graphics, PNG) or other formats; it should be noted that here is only The format of the road image is exemplified, and the embodiment of the present disclosure does not limit the format of the sample image.

In practical applications, sample images can be acquired from a local storage area or a network, or image acquisition devices can be used to acquire sample images.

Understandably, since the positive sample images include intersections, the training of the neural network based on the positive sample images is beneficial to enable the trained neural network to detect the intersections in the road images.

The training process of the above-mentioned neural network is exemplarily described below with reference to the accompanying drawings.

FIG. 2 is a flowchart of a neural network training method according to an embodiment of the present disclosure. As shown in FIG. 2 , the process may include:

Step 201: Perform feature extraction on the sample image to obtain a feature map of the sample image.

In this embodiment of the present disclosure, the feature map of the sample image may be used to represent at least one of the following features of the sample image: color feature, texture feature, shape feature, and spatial relationship feature; for the implementation of this step, exemplarily, the The sample image is input into the neural network, and the neural network is used to extract the feature of the sample image to obtain the feature map of the sample image.

In this embodiment of the present disclosure, the type of the neural network is not limited. Exemplarily, the neural network may be a Single Shot MultiBox Detector (SSD), You Only LookOnce (You Only LookOnce,), a fast Faster Region-Convolutional Neural Networks (Faster RCNN) or other deep learning-based neural networks. In the embodiments of the present disclosure, the network structure of the neural network is not limited. For example, the network structure of the neural network may be a 50-layer residual network structure, a VGG16 network structure, or a MobileNet network structure.

Step 202: Determine the detection result of the sample image according to the feature map of the sample image.

In the embodiment of the present disclosure, it can be determined whether the road shown in the sample image has an intersection according to the feature map of the sample image, and the detection result can be obtained. Obviously, the detection result includes the following two situations: the road shown in the sample image has an intersection, and the road shown in the sample image has an intersection, and the sample image has an intersection. There is no intersection on the road shown.

In the embodiment of the present disclosure, when the sample image is a positive sample image, the labeling result of the sample image is the labeling frame of the intersection on the road shown in the positive sample image, and the labeling frame represents the position of the intersection in the positive sample image, And the lower border of the above-mentioned marked frame is on the road surface of the road shown by the positive sample image; obviously, when the sample image is a positive sample image, the detection result of the sample image can be determined according to the feature map of the sample image, that is, the detection result of the junction can be determined. Check box.

Step 203: Adjust the network parameter value of the neural network according to the labeling result of the sample image and the detection result.

For the implementation of this step, for example, the network parameter value of the neural network can be adjusted according to the difference between the labeling result of the sample image and the above-mentioned detection result. In actual implementation, the loss of the neural network can be calculated, and the loss of the neural network is used to represent the difference between the labeling result of the sample image and the above detection result; then, the loss of the initial neural network can be reduced according to the loss of the initial neural network as The goal is to adjust the network parameter values of the neural network.

Step 204 : Determine whether the detection result of the sample image by the neural network after the adjustment of the network parameter value meets the set accuracy requirement, if not, return to step 201 ; if so, execute step 205 .

Here, the set accuracy requirement may be that the difference between the detection result of the sample image and the labeling result of the sample image is within a preset range.

Step 205: Use the neural network adjusted by the network parameter values as the trained neural network.

In practical applications, steps 201 to 205 may be implemented by a processor in an electronic device, and the above-mentioned processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. A sort of.

It can be seen that, in the embodiment of the present disclosure, in the training process of the neural network, the detection result of the sample image can be determined according to the feature map of the sample image; therefore, the trained neural network can be able to obtain clear Traffic lights or ground stop line images, or when there are no traffic lights or ground stop lines at intersections, intersection detection can also be achieved based on the feature map of road images; and, since the positive sample images include intersections, the neural network based on the positive sample images is used to detect the intersection. It is beneficial to enable the trained neural network to detect intersections in road images.

In practical applications, when marking the frame of intersections for positive sample images, since many intersections do not have obvious markers, there are also great difficulties in data labeling; for this problem, in the embodiments of the present disclosure , can be solved in a variety of ways, which are explained below with several examples.

In the first example, the lower border of the annotation box of the intersection on the road shown in the positive sample image is on the road surface of the road shown in the positive sample image; in this way, even when there are no obvious landmarks at the intersection, it can be determined that The lower border of the labeling box of the intersection is beneficial for labeling; further, since the labeling box marking the intersection is on the road surface, the labeling box of the intersection is consistent with the actual situation, and then the labeling box on the road shown in the sample image is on the road. On the basis of the annotation frame of the intersection, the trained neural network can accurately obtain the detection frame of the intersection.

On the basis of the first example, as an optional implementation, in the case where the positive sample image includes the stop line of the intersection of the road, the lower part of the labeling frame of the intersection on the road shown in the positive sample image The frame is aligned with the above-mentioned stop line; since the lower frame of the label box marking the intersection is aligned with the stop line, the label frame of the intersection is consistent with the actual situation, and then on the basis of the label frame of the intersection on the road shown in the sample image , so that the trained neural network can accurately obtain the detection frame of the intersection.

On the basis of the first example, as an optional implementation, the intersection of the positive sample image is marked with a rectangular frame. The lower border of the callout box is marked on the road, and the height of the rectangle callout box is set to a fixed value, for example, the height of the rectangle callout box is 80 pixels.

In the second example, the difference between the heights of the annotation boxes in the multiple positive sample images including the same intersection is within a preset range; the preset range can be preset according to the actual situation, for example, including multiple positive samples at the same intersection The heights of the annotation boxes in the images are the same, and they are all 80 pixels.

It can be seen that since the height difference of the labeling boxes of the intersections in the multiple positive sample images containing the same intersection is within the preset range, the consistency of the labeling boxes of the intersections of the multiple positive sample images can be guaranteed. On the basis of the annotation frame of the intersection of the image, it is beneficial to speed up the training process of the neural network.

In practical applications, multiple positive sample images containing the same intersection may be images captured continuously.

In the third example, in the positive sample image, when the intersection ahead can be identified, the annotation frame of the intersection needs to be marked.

In the fourth example, in the positive sample image, when there is severe occlusion or the naked eye cannot distinguish whether it is an intersection area, the intersection is not marked.

Optionally, when the sample image is a negative sample image, there is no intersection on the road in the negative sample image, and the annotation result of the sample image indicates that there is no annotation frame in the negative sample image.

It can be seen that by inputting the negative sample image into the neural network and performing the network training of the neural network, the error detection rate of the trained neural network for images that do not contain the intersection area can be reduced, that is, it can be more accurately detected. Contains an image of the intersection area.

In one embodiment, when the above-mentioned sample images include positive sample images and negative sample images, the ratio of the positive sample images to the negative sample images is greater than the set ratio threshold; in this way, by inputting enough positive sample images into the neural network, the The network training of the neural network can enable the trained neural network to more accurately detect the intersection area including the image of the intersection.

In the embodiment of the present disclosure, after the trained neural network is obtained, the road image can be input into the trained neural network, and the trained neural network can be used to detect intersections, and then determine the intersection on the road shown by the road image. The detection box, or, determines that there is no intersection on the road shown by the road image.

FIG. 3 is an example diagram of an embodiment of the present disclosure using a trained neural network for intersection detection. As shown in FIG. 3 , the image to be detected represents a road image captured by a single camera of a vehicle, and the detection network represents a trained neural network. It can be seen that the intersection detection result includes a detection frame representing the intersection, and the lower border of the intersection detection frame fits the road surface.

On the basis of the intersection detection method proposed in the foregoing embodiment, the embodiment of the present disclosure also proposes an intelligent driving method, which can be applied to an intelligent driving device. Here, the intelligent driving device includes but is not limited to automatic driving Vehicles with Advanced Driving Assistant System (ADAS), robots equipped with ADAS, etc.

FIG. 4 is a flowchart of an intelligent driving method according to an embodiment of the present disclosure. As shown in FIG. 4 , the flowchart may include:

Step 401: Obtain road images.

The implementation manner of this step has been described in the foregoing description, and will not be repeated here.

Step 402: Perform intersection detection on the road image according to any of the above-mentioned intersection detection methods.

Combining the above-mentioned contents, it can be seen that the detection result obtained by performing intersection detection on the road image can be a detection frame that determines the intersection on the road shown in the road image, or it can be determined that there is no intersection on the road shown in the road image. ; On the basis of determining the detection frame of the intersection, the distance between the device that collects the road image and the intersection can also be determined.

Step 403: Control the driving of the smart driving device according to the distance between the smart driving device that collects the road image and the intersection.

In practical applications, it is possible to directly control the driving of intelligent driving devices (autonomous driving and robots), or send instructions to the driver, and the driver can control the driving of the vehicle (such as a vehicle equipped with ADAS).

It can be seen that based on the intersection detection method, the distance between the intelligent driving device that collects road images and the intersection can be obtained, which is beneficial to provide assistance for vehicle driving according to the distance between the intelligent driving device that collects road images and the intersection. Safety of vehicle driving.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Intrinsic logical determination

On the basis of the intersection detection method proposed in the foregoing embodiments, the embodiment of the present disclosure proposes an intersection detection apparatus.

FIG. 5 is a schematic diagram of the composition and structure of an intersection detection apparatus according to an embodiment of the present disclosure. As shown in FIG. 5 , the apparatus includes: a first extraction module 501 , a detection module 502 and a first determination module 503 , wherein,

a first extraction module 501, configured to perform feature extraction on a road image to obtain a feature map of the road image;

The detection module 502 is configured to determine the detection frame of the intersection on the road shown in the road image according to the feature map of the road image; the detection frame of the intersection represents the area of the intersection in the road image, the The lower border of the detection frame of the intersection is on the road surface of the road;

The first determination module 503 is configured to determine the distance between the device that collects the road image and the intersection according to the lower border of the detection frame of the intersection.

Optionally, the detection module 502 is further configured to determine, according to the feature map of the road image, that there is no intersection on the road shown by the road image.

Optionally, the first determination module 503 is configured to convert the coordinates according to the position of the lower border of the detection frame of the intersection in the road image and the plane of the road image and the road surface of the road. determine the position of the lower border of the detection frame of the intersection on the road; according to the position of the lower border of the detection frame of the intersection on the road and the device that collects the road image on the road position, and obtain the distance between the device that collects the road image and the intersection.

Optionally, the device is implemented based on a neural network, and the neural network is obtained by training a sample image and an annotation result of the sample image, and the annotation result of the sample image includes the annotation of the intersection on the road shown in the positive sample image. frame, the labeling frame represents the position of the intersection in the positive sample image, and the lower border of the labeling frame is on the road surface of the road shown in the positive sample image.

In practical applications, the first extraction module 501, the detection module 502 and the first determination module 503 can all be implemented by using a processor in an electronic device, and the above-mentioned processor can be an ASIC, DSP, DSPD, PLD, FPGA, CPU, controller , at least one of a microcontroller and a microprocessor.

FIG. 6 is a schematic structural diagram of a neural network training apparatus according to an embodiment of the present disclosure. As shown in FIG. 6 , the apparatus may include a second extraction module 601, a second determination module 602, and an adjustment module 603, wherein,

The second extraction module 601 is configured to perform feature extraction on the sample image to obtain a feature map of the sample image;

A second determination module 602, configured to determine the detection result of the sample image according to the feature map of the sample image;

An adjustment module 603, configured to adjust the network parameter value of the neural network according to the labeling result of the sample image and the detection result;

When the sample image is a positive sample image, the labeling result of the sample image is the labeling frame of the intersection on the road shown by the positive sample image, and the labeling frame represents the position of the intersection in the positive sample image , and the lower border of the labeling frame of the intersection on the road shown in the positive sample image is on the road surface of the road shown in the positive sample image.

Optionally, the positive sample image includes a stop line of a road intersection, and the lower border of the label box of the intersection on the road shown in the positive sample image is aligned with the stop line.

Optionally, the difference between the heights of the annotation boxes in the multiple positive sample images including the same intersection is within a preset range.

Optionally, when the sample image is a negative sample image, there is no intersection on the road in the negative sample image, and the annotation result of the sample image includes that there is no annotation frame in the negative sample image.

In practical applications, the second extraction module 601, the second determination module 602 and the adjustment module 603 can all be implemented by using a processor in an electronic device, and the above-mentioned processor can be an ASIC, DSP, DSPD, PLD, FPGA, CPU, controller , at least one of a microcontroller and a microprocessor.

FIG. 7 is a schematic structural diagram of an intelligent driving device according to an embodiment of the present disclosure. As shown in FIG. 7 , the device includes: an acquisition module 701 and a processing module 702 , wherein,

an acquisition module 701, configured to acquire road images;

The processing module 702 is configured to perform intersection detection on the road image according to any of the above-mentioned intersection detection methods; and control the driving of the device according to the distance between the intelligent driving device that collects the road image and the intersection.

In practical applications, both the acquisition module 701 and the processing module 702 can be implemented by a processor in an intelligent driving device, and the above-mentioned processor can be an ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, or microprocessor. at least one of them.

In addition, each functional module in this embodiment may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software function modules.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or Said part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, removable hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Specifically, the computer program instructions corresponding to any intersection detection method, neural network training method, or intelligent driving method in this embodiment can be stored on a storage medium such as an optical disk, a hard disk, and a U disk. When the computer program instructions corresponding to any intersection detection method, neural network training method or intelligent driving method are read or executed by an electronic device, any intersection detection method, neural network training method or intelligent driving method of the foregoing embodiment is realized. method.

Based on the same technical concept as the foregoing embodiments, see FIG. 8 , which shows an electronic device 80 provided by an embodiment of the present disclosure, which may include: a memory 81 and a processor 82 ; wherein,

The memory 81 is used to store computer programs and data;

The processor 82 is configured to execute the computer program stored in the memory, so as to implement any one of the intersection detection methods, neural network training methods or intelligent driving methods in the foregoing embodiments.

In practical applications, the above-mentioned memory 81 may be a volatile memory (volatile memory), such as RAM; or a non-volatile memory (non-volatile memory), such as ROM, flash memory (flash memory), hard disk (Hard Disk Drive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and provide instructions and data to processor 82.

The above-mentioned processor 82 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It can be understood that, for different devices, the electronic device used to implement the function of the processor may also be other, which is not specifically limited in the embodiment of the present disclosure.

In some embodiments, the functions or modules included in the apparatuses provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the descriptions of the above method embodiments. For brevity, here No longer.

The above description of the various embodiments tends to emphasize the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, details are not repeated herein.

The methods disclosed in each method embodiment provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment.

The features disclosed in each product embodiment provided in this application can be combined arbitrarily without conflict to obtain a new product embodiment.

The features disclosed in each method or device embodiment provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment or device embodiment.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD), including several instructions to make a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present invention.

The embodiments of the present invention have been described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of the present invention, without departing from the scope of protection of the present invention and the claims, many forms can be made, which all belong to the protection of the present invention.

Claims (10)

1. An intersection detection method, characterized in that the method comprises:

carrying out feature extraction on a road image to obtain a feature map of the road image;

determining a detection frame of an intersection on a road shown by the road image according to the characteristic diagram of the road image; the detection frame of the intersection represents the area of the intersection in the road image, and the lower frame of the detection frame of the intersection is on the road surface of the road;

and determining the distance between the equipment for acquiring the road image and the intersection according to the lower frame of the detection frame of the intersection.

2. The method of claim 1, further comprising:

and determining that the road shown by the road image has no intersection according to the characteristic diagram of the road image.

3. The method of claim 1, wherein determining the distance between the device for acquiring the road image and the intersection according to the lower border of the detection frame of the intersection comprises:

determining the position of the lower frame of the detection frame of the intersection on the road according to the position of the lower frame of the detection frame of the intersection in the road image and the coordinate conversion relation between the plane of the road image and the road surface of the road;

and obtaining the distance between the equipment for acquiring the road image and the intersection according to the position of the lower frame of the detection frame of the intersection on the road and the position of the equipment for acquiring the road image on the road.

4. A neural network training method, comprising:

carrying out feature extraction on a sample image to obtain a feature map of the sample image;

determining the detection result of the sample image according to the characteristic diagram of the sample image;

adjusting the network parameter value of the neural network according to the labeling result and the detection result of the sample image;

when the sample image is a positive sample image, the labeling result of the sample image is a labeling frame of the intersection on the road shown by the positive sample image, the labeling frame represents the position of the intersection in the positive sample image, and the lower frame of the labeling frame of the intersection on the road shown by the positive sample image is on the road surface of the road shown by the positive sample image.

5. An intelligent driving method, comprising:

acquiring a road image;

the method according to any one of claims 1-3, wherein intersection detection is performed on the road image;

and controlling the equipment to run according to the distance between the intelligent running equipment for collecting the road image and the intersection.

6. The intersection detection device is characterized by comprising a first extraction module, a detection module and a first determination module; wherein,

the first extraction module is used for extracting the characteristics of the road image to obtain a characteristic map of the road image;

the detection module is used for determining a detection frame of an intersection on a road shown by the road image according to the characteristic diagram of the road image; the detection frame of the intersection represents the area of the intersection in the road image, and the lower frame of the detection frame of the intersection is on the road surface of the road;

and the first determining module is used for determining the distance between the equipment for acquiring the road image and the intersection according to the lower frame of the detection frame of the intersection.

7. An apparatus for neural network training, the apparatus comprising: a second extraction module, a second determination module, and an adjustment module, wherein,

the second extraction module is used for extracting the characteristics of the sample image to obtain a characteristic diagram of the sample image;

the second determining module is used for determining the detection result of the sample image according to the feature map of the sample image;

the adjusting module is used for adjusting the network parameter value of the neural network according to the labeling result and the detection result of the sample image;

when the sample image is a positive sample image, the labeling result of the sample image is a labeling frame of the intersection on the road shown by the positive sample image, the labeling frame represents the position of the intersection in the positive sample image, and the lower frame of the labeling frame of the intersection on the road shown by the positive sample image is on the road surface of the road shown by the positive sample image.

8. An intelligent travel apparatus, characterized in that the apparatus comprises: an acquisition module and a processing module, wherein,

the acquisition module is used for acquiring a road image;

a processing module for performing intersection detection on the road image according to the method of any one of claims 1 to 3; and controlling the equipment to run according to the distance between the intelligent running equipment for collecting the road image and the intersection.

9. An electronic device comprising a processor and a memory for storing a computer program operable on the processor; wherein,

the processor is configured to run the computer program to execute the intersection detection method according to any one of claims 1 to 3, the neural network training method according to claim 4, or the intelligent driving method according to claim 5.

10. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the intersection detection method of any one of claims 1 to 3 or the neural network training method of claim 4 or the intelligent driving method of claim 5.

Images