CN118629216B - Road target recognition method and system based on radar and vision fusion - Google Patents

Abstract

The application provides a road target identification method and a road target identification system based on a thunder fusion, which are characterized in that firstly, road target tracking characteristics of a thunder fusion data stream are extracted, potential risk targets are accurately identified based on the road target tracking characteristics, the monitoring range is effectively narrowed, the potential risk indexes of the targets are further calculated by acquiring tracking state path data of reference potential risk targets and combining the tracking state path data with global road target tracking characteristics, the potential risk indexes not only reflect the instant dangerous degree of the targets, but also disclose the abnormal behavior trend of the targets, and finally, the road target identification data generated based on the potential risk indexes provides visual and quantized risk assessment results for traffic management departments, and is beneficial to quick response and scientific scheduling, so that the safety and the traffic efficiency of road traffic are greatly improved. Therefore, the road target is efficiently and accurately identified and risk assessment is realized through the radar fusion technology and the refined risk assessment model.

Description

Road target recognition method and system based on radar and vision fusion

Technical Field

The present application relates to the field of artificial intelligence technology, and more specifically, to a road target recognition method and system based on radar-vision fusion.

Background Art

With the acceleration of urbanization and the continuous growth of the number of motor vehicles, urban traffic management faces unprecedented challenges. Problems such as road congestion and frequent traffic accidents have become increasingly prominent, posing a serious threat to public travel safety and urban operation efficiency. Traditional traffic monitoring methods mainly rely on single sensor technology, such as ground sensing coils and video cameras, which have obvious limitations in target detection, tracking and risk assessment. For example, ground sensing coils are complex to install and have a limited sensing range, while video cameras are easily affected by environmental factors such as lighting and weather, resulting in high false alarm and missed alarm rates.

In order to meet these challenges, radar-visual fusion technology has gradually become a research hotspot in the field of intelligent transportation in recent years. By combining the advantages of radar and visual sensors, radar-visual fusion technology achieves all-round and high-precision perception of road targets. Radar sensors can accurately measure the position, speed and other information of the target without being affected by environmental factors such as light and haze; while visual sensors can provide rich target image information, which helps to identify the type and specific behavior of the target. Fusion processing of the two data can not only make up for the shortcomings of a single sensor, but also significantly improve the accuracy and robustness of road target recognition.

However, existing methods for road target recognition based on radar and vision fusion mostly focus on static detection and tracking of targets, but lack dynamic evaluation and prediction of potential risk targets. In actual traffic scenarios, the identification and evaluation of potential risk targets are of vital importance for preventing traffic accidents and optimizing traffic flow. Therefore, it is particularly important to develop a method that can accurately identify potential risk targets in real time and evaluate their potential risk index.

Summary of the invention

In view of the above-mentioned problems, in combination with the first aspect of the present application, an embodiment of the present application provides a road target recognition method based on radar-visual fusion, the method comprising:

Acquire a road target tracking feature of a radar-vision fusion data stream, wherein the road target tracking feature of the radar-vision fusion data stream reflects a content representation of the radar-vision fusion data stream;

Based on the road target tracking characteristics of the radar and visual fusion data stream, determining at least one reference potential risk target in the radar and visual fusion data stream;

Acquire the tracking state path data of each of the reference potential risk targets, and determine the potential risk index of each of the reference potential risk targets based on the tracking state path data of each of the reference potential risk targets and the road target tracking characteristics of the radar-visual fusion data stream, wherein the potential risk index reflects the abnormal behavior trend parameter of the reference potential risk target in the radar-visual fusion data stream;

Based on the potential risk index of each of the reference potential risk targets, the road target recognition data of the radar and visual fusion data stream is determined.

On the other hand, an embodiment of the present application also provides a road target recognition system based on radar and vision fusion, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

Based on the above aspects, the embodiment of the present application significantly improves the accuracy and real-time performance of road target recognition by fusing the data of radar and visual sensors. First, the road target tracking features of the radar-visual fusion data stream are extracted to comprehensively characterize the dynamic changes of the traffic scene, providing rich basic information for subsequent risk assessment. Then, based on these road target tracking features, potential risk targets are accurately identified, effectively narrowing the monitoring scope, so that the traffic management department can focus on high-risk areas or objects. By obtaining the tracking state path data of the reference potential risk target and combining it with the global road target tracking features, the potential risk index of each target is further calculated. The potential risk index not only reflects the immediate danger level of the target, but also reveals its abnormal behavior trend, providing a scientific basis for predicting and preventing traffic accidents. Finally, the road target recognition data generated based on the potential risk index provides the traffic management department with intuitive and quantitative risk assessment results, which is helpful for rapid response and scientific scheduling, thereby greatly improving the safety and traffic efficiency of road traffic. Therefore, through the radar-visual fusion technology and the refined risk assessment model, efficient and accurate identification and risk assessment of road targets are achieved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the execution flow of a road target recognition method based on radar-visual fusion provided in an embodiment of the present application.

DETAILED DESCRIPTION

The present application will be described in detail below in conjunction with the accompanying drawings in the specification. FIG1 is a flow chart of a road target recognition method based on radar-visual fusion provided by an embodiment of the present application. The road target recognition method based on radar-visual fusion will be introduced in detail below.

Step S110 , obtaining a road target tracking feature of a radar and visual fusion data stream, wherein the road target tracking feature of the radar and visual fusion data stream reflects a content representation of the radar and visual fusion data stream.

In this embodiment, an advanced traffic monitoring system is deployed in a busy urban traffic network. The traffic monitoring system integrates radar and visual sensors, and can collect and fuse the tracking data of road targets in real time. These data are organized into radar-visual fusion data streams and transmitted to the server for processing and analysis. The main task of the server is to identify potential risk targets from these radar-visual fusion data and evaluate their potential risk index so that the traffic management department can take timely measures to prevent traffic accidents.

In detail, the server first accesses the radar-visual fusion data stream, which contains real-time data from radar and visual sensors. These data are preprocessed and fused to form tracking data containing rich road target information. The server extracts road target tracking features by parsing these data, and these road target tracking features constitute a first feature vector set.

In one example, a server receives radar-visual fusion data streams from a traffic monitoring system through a network interface. These radar-visual fusion data streams are transmitted in the form of continuous time series, containing multiple data frames per second, and each data frame contains detailed tracking information of road targets. For each data frame, the server runs feature extraction algorithms, which can automatically extract key features of road targets from radar data and visual data, such as vehicle speed, acceleration, driving direction, position coordinates, vehicle size, and pedestrian walking speed and walking direction. These features are encoded into vector form and combined into a first feature vector set. The first feature vector set not only contains the features of a single road target, but also reflects the global content representation of the entire radar-visual fusion data stream by integrating the features of multiple road targets. For example, the server can further describe the overall state of the traffic flow by calculating global statistics such as average speed, maximum speed difference, and target density.

Step S120: determining at least one reference potential risk target in the radar and visual fusion data stream based on the road target tracking characteristics of the radar and visual fusion data stream.

In this embodiment, after acquiring the road target tracking features, the server further analyzes these road target tracking features to determine potential risk targets in the radar-visual fusion data stream.

In detail, the server runs pattern recognition algorithms (see the subsequent road target recognition network section), which can identify reference potential risk targets that meet specific risk patterns. For example, some high-risk behavior patterns can be learned based on historical accident data, such as rapid lane changes, driving in the wrong direction, pedestrians crossing the road, etc. When the server detects similar behavior patterns in the real-time data stream, it will mark these targets as potential risk targets. In addition, road targets that deviate from normal traffic behavior patterns can also be identified. From this, the server generates a reference potential risk target list, which contains all road targets marked as potential risks.

Step S130, obtaining the tracking status path data of each of the reference potential risk targets, and determining the potential risk index of each of the reference potential risk targets based on the tracking status path data of each of the reference potential risk targets and the road target tracking characteristics of the radar-visual fusion data stream, wherein the potential risk index reflects the abnormal behavior trend parameters of the reference potential risk target in the radar-visual fusion data stream.

In this embodiment, for each reference potential risk target, the server retrieves its tracking state path data, which describes the motion trajectory and state changes of the reference potential risk target over the past period of time, such as speed changes, direction changes, acceleration, etc.

The server can calculate the matching degree between the tracking state path data of each potential risk target and the first feature vector set. The matching degree calculation may involve multiple factors, such as the deviation of the target's speed from the global average speed, whether the target's acceleration is abnormal, whether the target frequently changes the driving direction, etc. The server quantifies these factors into specific numerical indicators and calculates the matching degree by weighted summation and other methods. Based on the matching degree calculation results, the server assigns a potential risk index to each potential risk target. The potential risk index reflects the abnormal behavior trend parameters of the target in the radar-vision fusion data stream, that is, the possibility of the target causing a traffic accident or traffic congestion. The calculation of the potential risk index can adopt a variety of methods such as linear models, nonlinear models or machine learning models.

For example, suppose the server detects that a vehicle has been accelerating over the past 5 seconds and has frequently changed its direction. The server first extracts the tracking state path data of the vehicle, including the speed sequence and the direction change sequence. The server then calculates the matching degree of these data with the first feature vector set. During the calculation process, the server finds that the speed deviation of the vehicle is significantly higher than the global average speed deviation, and the frequency of direction changes is also much higher than that of normal vehicles. Therefore, the server assigns a higher potential risk index to the vehicle, indicating that it has a higher risk of traffic accidents.

Step S140, determining the road target recognition data of the radar and visual fusion data stream based on the potential risk index of each reference potential risk target.

Finally, the server generates road target recognition data based on the potential risk index of each reference potential risk target. The road target recognition data includes the recognition results of all road targets in the radar-vision fusion data stream, as well as the recognition results and potential risk index of the potential risk targets.

Specifically, the server integrates the recognition results of all road targets to form a comprehensive road target recognition report. The road target recognition report contains basic information such as the type (such as vehicle, pedestrian, bicycle, etc.), location, speed, and driving direction of each road target.

For those targets with higher potential risk index, the server will highlight them in the identification report. These targets may be marked in red or highlighted so that the traffic management department can quickly notice them. The server transmits the road target identification data to the traffic management department or related application platform in real time. These road target identification data can be sent in a streaming manner through the network interface to ensure that the traffic management department can obtain the latest traffic information in a timely manner.

Based on the above steps, the embodiment of the present application significantly improves the accuracy and real-time performance of road target recognition by fusing the data of radar and visual sensors. First, the road target tracking features of the radar-visual fusion data stream are extracted to comprehensively characterize the dynamic changes of the traffic scene, providing rich basic information for subsequent risk assessment. Then, based on these road target tracking features, potential risk targets are accurately identified, effectively narrowing the monitoring scope, so that the traffic management department can focus on high-risk areas or objects. By obtaining the tracking state path data of the reference potential risk target and combining it with the global road target tracking features, the potential risk index of each target is further calculated. The potential risk index not only reflects the immediate danger level of the target, but also reveals its abnormal behavior trend, providing a scientific basis for predicting and preventing traffic accidents. Finally, the road target recognition data generated based on the potential risk index provides intuitive and quantitative risk assessment results for the traffic management department, which is helpful for rapid response and scientific scheduling, thereby greatly improving the safety and traffic efficiency of road traffic. Therefore, through the radar-visual fusion technology and the refined risk assessment model, efficient and accurate identification and risk assessment of road targets are achieved.

In a possible implementation manner, the road target tracking feature of the radar and visual fusion data stream includes a first feature vector set, and the first feature vector set reflects the global content representation of the radar and visual fusion data stream.

Step S130 includes: determining a potential risk index of each of the reference potential risk targets based on a matching degree between the tracking state path data of each of the reference potential risk targets and the first feature vector set.

In an intelligent traffic management system, radar and vision sensors are integrated and deployed at major traffic nodes in the city, such as intersections, highway entrances, etc. Radar sensors can accurately measure the distance, speed and direction of targets (such as vehicles and pedestrians), while vision sensors provide detailed image information of the target, including type, color, size, etc.

The server receives the raw data from these sensors in real time and performs fusion processing on them. The fusion process not only combines the advantages of radar and visual sensors, but also removes redundant information and noise through complex algorithms to generate high-quality radar-visual fusion data streams.

In the radar-vision fusion data stream, the tracking features of road targets are abstracted and encoded into a first feature vector set, which is a multi-dimensional data structure. Each dimension represents a specific feature, such as the target's speed, acceleration, position coordinates, driving direction, type (vehicle, pedestrian), size, etc. These feature vectors together constitute a global content representation of the current traffic scene.

For example, at a busy intersection, the radar-visual fusion data stream received by the server contains multiple vehicles in motion and a pedestrian who is about to cross the road. For each vehicle, the server generates a feature vector based on its radar and visual data. The feature vector contains the vehicle's speed, acceleration, current position (latitude and longitude), direction of travel (east, south, west, north or diagonal), vehicle type (sedan, truck, bus, etc.) and possible size information. Similarly, for pedestrians, the server also generates a feature vector containing information such as walking speed, direction, and position. All these feature vectors are organized into a first feature vector set, which fully reflects the current traffic conditions at the intersection.

After obtaining the first feature vector set, the server begins to analyze these first feature vector sets to identify potential traffic risks. The server first selects reference potential risk targets from the radar-visual fusion data stream according to certain rules or algorithms (such as pattern recognition based on historical accident data). These targets can be vehicles with abnormal driving trajectories, vehicles with too fast or too slow speeds, vehicles that violate traffic rules (such as driving against traffic or running red lights), or pedestrians in dangerous positions (such as pedestrians about to enter the blind spot of a vehicle).

For each reference potential risk target, the server retrieves its tracking state path data, which records the target's movement trajectory and state changes over the past period of time, such as speed changes, direction changes, position movements, etc. The server compares and analyzes these tracking state path data with the first feature vector set to assess the potential risk of the target.

Specifically, the server calculates the matching degree between the tracking state path data of each reference potential risk target and the first feature vector set. The matching degree calculation takes into account multiple factors, including but not limited to:

Speed Deviation: The degree of deviation between the target's current speed and the global average speed or the typical speed of this type of target.

Abnormal acceleration: Whether the target acceleration increases or decreases suddenly, which may indicate emergency braking or acceleration.

Direction changes: Whether the target's driving direction changes frequently or turns suddenly, which may increase the risk of collision.

The relationship between location and traffic flow: whether the target's current location is in a high-risk area such as a traffic-intensive area, intersection, or zebra crossing.

Traffic rules compliance: whether the target violates traffic rules, such as driving in the wrong direction, running a red light, etc.

The server quantifies these factors into specific numerical indicators and assigns a certain weight to each indicator. Then, the matching degree is calculated by weighted summation. The higher the matching degree, the greater the deviation between the target's tracking state path and the global traffic flow or typical behavior pattern, and the higher its potential risk.

Based on the matching calculation results, the server assigns a potential risk index to each reference potential risk target. The potential risk index is a quantitative risk assessment result that reflects the probability of the target causing a traffic accident or traffic congestion in the current traffic scenario. The potential risk index can be standardized as needed to facilitate comparison and analysis between different times and scenarios.

For example, in the previous intersection scenario, the server identified a vehicle that was approaching the zebra crossing quickly and without slowing down as a reference potential risk target. The server retrieved the tracking status path data of the vehicle and found that its speed was much higher than the global average speed and its acceleration was large, and its driving direction was directly pointing to the zebra crossing where pedestrians were crossing. The server compared and analyzed these data with the first feature vector set and calculated a higher matching degree. Based on the matching result, the server assigned a higher potential risk index to the vehicle and immediately sent this information to the traffic management department or relevant emergency response agency so that timely measures can be taken to prevent traffic accidents.

In a possible implementation, step S140 includes: outputting the reference potential risk target whose potential risk index meets the first setting requirement as the potential risk target included in the radar-visual fusion data stream. The first setting requirement includes that the potential risk index is not less than a first threshold value, or arranging each of the reference potential risk targets in descending order of the potential risk index, and arranging them in the first i order of the arrangement result, where i is a set positive integer.

After obtaining the potential risk indexes of all reference potential risk targets, the server needs to determine which targets are the potential risk targets that really need attention in the current radar-vision fusion data stream based on these potential risk indexes. This step involves screening and sorting the potential risk indexes to output road target recognition data that meets specific conditions.

The server first sets the first setting requirements according to actual needs. These first setting requirements may be determined based on a variety of factors such as traffic rules, historical accident data, current traffic flow, etc. In this scenario, it is assumed that the first setting requirements set by the server have two conditions: one is that the potential risk index is not less than the first threshold value (for example, the first threshold value is set to 0.7, indicating that only targets with a potential risk index higher than or equal to 0.7 will be considered high-risk targets); the second is that targets in the first i order of the arrangement result are also considered high-risk targets according to the descending order of the potential risk index (for example, setting i to 5 means that even if the potential risk index of some targets is lower than the first threshold value, as long as their risk index ranks in the top five among all targets, they will also be paid attention to).

The server traverses all reference potential risk targets and their corresponding potential risk indexes. For each reference potential risk target, the server first checks whether its potential risk index is not less than the first threshold value (0.7). If so, the target is marked as a high-risk target and is prepared to be included in the final road target identification data. If the potential risk index of a reference potential risk target is less than the first threshold value, the server will not exclude it immediately, but will continue to consider its potential risk index ranking among all targets.

The server sorts all reference potential risk targets in descending order of potential risk index, which means that the targets with the highest potential risk will be at the front of the list. After the sorting is completed, the server intercepts the targets in the top i order in the sorting result (the top 5 in this case). These targets are also considered high-risk targets because their risk index ranks high among all targets, even though their potential risk index may be lower than the first threshold value. The server organizes the high-risk targets determined after screening and sorting (including those targets whose potential risk index is not less than the first threshold value, and the targets ranked in the top i) into road target identification data, which can include detailed information such as the target's unique identifier, type (such as vehicle, pedestrian), location coordinates, speed, driving direction, potential risk index, etc.

The server transmits these road target recognition data to the traffic management department or related application platforms in real time so that they can take timely measures to prevent traffic accidents.

For example, suppose that the server identifies 10 reference potential risk targets when processing a certain radar-visual fusion data stream and calculates their potential risk index. After screening and sorting, the server finds that there are three potential risk targets with potential risk indexes not less than 0.7, namely target A (index 0.8), target B (index 0.75) and target C (index 0.7). Among the remaining targets, the top two are target D (index 0.65, ranked 4th) and target E (index 0.6, ranked 5th) in descending order of potential risk index. If the server sets the i value to 5, the final output road target recognition data will include targets A, B, C, D and E, because they either have a potential risk index not less than 0.7 or are in the top 5 in the potential risk index ranking. These data will be transmitted to the traffic management department in real time so that they can pay attention to these high-risk targets and take corresponding measures.

In a possible implementation, step S140 includes: outputting the reference potential risk target whose potential risk index meets the second setting requirement as a significant potential risk target included in the radar-visual fusion data stream. The second setting requirement includes: the potential risk index is not less than a second threshold value, or arranging each of the reference potential risk targets in descending order of the potential risk index, and being located in the first j order of the arrangement result, where j is a set positive integer.

In this embodiment, after obtaining the potential risk indexes of all reference potential risk targets, the server will screen significant potential risk targets according to the second setting requirements. These targets have extremely high risks in the current traffic flow and need to be paid special attention.

The server sets the second set requirements according to the needs of the traffic management department or the safety standards preset by the system. These requirements are usually more stringent than the first set requirements to ensure that the significant potential risk targets screened out are truly highly risky. In this scenario, it is assumed that the second set requirements set by the server include two conditions: one is that the potential risk index is not less than the second threshold value (for example, the second threshold value is set to 0.9, which is much higher than the first threshold value of 0.7, indicating that only targets with extremely high potential risk indexes will be considered as significant potential risk targets); the second is that targets ranked in the first j order of the arrangement result are also considered as significant potential risk targets (for example, setting j to 3 means that even if the potential risk index of some targets is slightly lower than the second threshold value, as long as their risk index ranks in the top three among all targets, they will also receive special attention).

The server first traverses all reference potential risk targets and their corresponding potential risk indexes. For each potential risk target, the server checks whether its potential risk index is not less than the second threshold value (0.9). If so, the target is immediately marked as a significant potential risk target. If the potential risk index of a potential risk target is less than the second threshold value, the server will not exclude it immediately, but will include it in the subsequent sorting and interception process.

The server sorts all reference potential risk targets in descending order of potential risk index to ensure that the targets with the highest potential risk are at the top of the list. After the sorting is completed, the server intercepts the targets in the first j order of the sorting results (the first 3 in this example). Even if the potential risk index of these targets may be slightly lower than the second threshold value, they are also considered as significant potential risk targets because their risk index ranks very high among all targets.

The server will sort the significant potential risk targets identified after screening and sorting into significant potential risk target data, which usually contains key information such as the target's unique identifier, type (such as vehicle, pedestrian), location coordinates, speed, driving direction, potential risk index, etc. The server will transmit these significant potential risk target data to the traffic management department or relevant emergency response agencies in real time, and these data will serve as an important basis for the traffic management department to formulate response measures and dispatch resources.

For example, suppose the server identifies multiple reference potential risk targets when processing a certain radar fusion data stream and calculates their potential risk indexes. After screening and sorting, the server finds that:

There are two potential risk targets whose potential risk index is not less than 0.9, namely target F (index 0.95) and target G (index 0.92). These two potential risk targets are immediately marked as significant potential risk targets.

Among the remaining targets, in descending order of potential risk index, the first one is target H (index 0.88, ranking 3rd). Although its potential risk index is lower than the second threshold value of 0.9, it is also considered a significant potential risk target because it ranks third among all targets.

Ultimately, the significant potential risk target data output by the server will include detailed information on targets F, G, and H, which will be transmitted to the traffic management department in real time so that they can immediately pay attention to these high-risk significant potential risk targets and take corresponding emergency measures to prevent possible traffic accidents.

In a possible implementation, step S120 includes:

Step S121, based on the road target tracking characteristics of the radar and vision fusion data stream, determining the road event label information of the radar and vision fusion data stream, wherein the road event label information of the radar and vision fusion data stream reflects the label knowledge point information corresponding to the road event of the radar and vision fusion data stream.

Step S122: determining at least one reference potential risk target in the radar and vision fusion data stream based on the road target tracking characteristics of the radar and vision fusion data stream and the road event label information of the radar and vision fusion data stream.

In this embodiment, the server first receives and processes the radar-vision fusion data stream, extracts road target tracking features, such as the vehicle's speed, acceleration, position, and driving direction, as well as the pedestrian's walking speed and direction, etc. These road target tracking features are fused to form a comprehensive description of the current traffic scene.

Next, the server can automatically identify road events hidden in the data stream based on technologies such as deep learning, machine learning or rule engines. Specifically, it considers multiple factors such as the target's movement trajectory, speed change, position relationship, etc., as well as historical accident data and traffic rules knowledge base.

When a road event is identified, the server will assign one or more road event tags to it according to the type and characteristics of the event. These road event tags are predefined and represent different types of road events, such as "vehicle emergency stop", "pedestrian crossing", "traffic accident scene", etc. Each tag is associated with a series of knowledge point information, describing the possible consequences and response strategies of the road event. In order to enrich the information content of the tag, the server may query a tag knowledge base, which contains detailed descriptions of various road event tags, historical cases, response measures and other knowledge point information. By querying the knowledge base, the server can add more contextual information and practical suggestions for each identified road event tag.

Next, the server associates the road target tracking features with the road event tag information, which takes into account the relationship between the target's position, speed, driving direction and event tags. For example, if a vehicle target is approaching an area marked as "pedestrian crossing" and its speed is high and there is no sign of slowing down, then the vehicle may be considered a potential risk target.

Next, the server combines the road target tracking features and road event label information to evaluate the potential risk of each potential risk target through complex calculation logic. As a result, the server selects targets with higher potential risk index as reference potential risk targets. These reference potential risk targets have higher risks in the current traffic scenario and need special attention.

For example, suppose that in the radar-vision fusion data stream at a certain intersection, the server detects a "pedestrian crossing" event and assigns it a corresponding road event label. At the same time, the server also tracks multiple vehicles in motion. One of the vehicles (target X) is rapidly approaching the pedestrian crossing area and shows no signs of slowing down. The server finds through association analysis that there is a high-risk relationship between target X and the "pedestrian crossing" event label, and calculates that the potential risk index of target X is high. Therefore, the server selects target X as a reference potential risk target and records it together with the "pedestrian crossing" event label in the potential risk target list, which is then transmitted to the traffic management department in real time so that they can take timely measures to prevent potential traffic accidents.

In a possible implementation, step S121 includes:

Step S1211, based on the road target tracking characteristics of the radar-visual fusion data stream, determine the confidences corresponding to the z road event labels respectively, and the confidence corresponding to the x-th road event label among the z road event labels reflects that the probability x that the radar-visual fusion data stream matches the x-th road event label is not greater than z.

Step S1212: based on the confidences respectively corresponding to the z road event labels, y road event labels with the largest confidences are selected from the z road event labels to generate y target road event labels.

Step S1213, based on the confidences corresponding to the y target road event labels, respectively, the label knowledge features corresponding to the y target road event labels are fused and calculated to generate a global label knowledge feature, and the road event label information of the radar-visual fusion data stream includes the global label knowledge feature.

In this embodiment, the server runs a pre-trained fully connected output unit, which is built based on deep learning, machine learning and other technologies and can identify various types of road events. The fully connected output unit receives road target tracking features as input, and outputs z road event labels and their corresponding confidences, which reflect the probability that the radar-visual fusion data stream matches each road event label. For the xth road event label (x is not greater than z), its confidence is obtained through complex calculation logic inside the fully connected output unit, which may include feature weighting, nonlinear transformation, decision boundary judgment and other steps, and finally outputs a value between 0 and 1 as the confidence. The closer the confidence is to 1, the higher the probability that the data stream matches the event label.

The server sorts the z road event labels in descending order according to their corresponding confidence levels. The sorted list reflects the degree of match between each event label and the current radar-vision fusion data stream, with the label with the highest confidence level at the front of the list. Based on the preset y value (y is a set positive integer, and y is less than or equal to z), the server selects the first y road event labels from the sorted list as target road event labels. These labels have the highest confidence levels in the current data stream and are therefore considered to be the most likely road events.

For each selected target road event label, the server queries a predefined label knowledge base, which contains detailed descriptions of each event label, historical cases, countermeasures, related knowledge points and other information. The server retrieves the knowledge features associated with the target label. The server has designed a set of feature fusion algorithms that can integrate the knowledge features of multiple target road event labels. The fusion calculation may involve multiple operations such as feature weighting, feature splicing, and feature transformation, aiming to extract global features that can fully reflect the risk status of the current traffic scene. After the feature fusion calculation, the server generates a global label knowledge feature, which not only contains the key information of each target road event label, but also removes redundancy and noise through the fusion algorithm, forming a comprehensive description of the road event in the current radar-vision fusion data stream.

Ultimately, the server outputs the global label knowledge features as road event label information of the radar-vision fusion data stream. This information will serve as an important basis for subsequent risk assessment, target identification, emergency response and other tasks.

For example, suppose that in the radar-visual fusion data stream of a certain intersection, the server identified five potential road event labels (z=5) through the fully connected output unit and calculated their respective confidences. These labels include "vehicle emergency stop", "pedestrian crossing", "vehicle driving in the wrong direction", "traffic congestion" and "traffic accident scene". After confidence sorting and selection, the server determined the three labels with the highest confidence as the target road event labels (y=3), namely "pedestrian crossing" (confidence 0.9), "vehicle emergency stop" (confidence 0.8) and "traffic congestion" (confidence 0.7).

Next, the server queries the label knowledge base and obtains the knowledge features associated with the three labels. Then, through feature fusion calculation, the server generates a global label knowledge feature that comprehensively reflects the high-risk state of pedestrian crossing, vehicle emergency stop and traffic congestion at the current intersection. Finally, the server outputs the global label knowledge feature as road event label information for reference by traffic management departments or emergency response agencies.

In a possible implementation manner, the road target tracking feature of the radar and vision fusion data stream includes a set of feature vectors corresponding to a plurality of radar and vision fusion data segments in the radar and vision fusion data stream.

Step S122 includes:

Step S1221, respectively aggregating the feature vector sets corresponding to each of the thunder and vision fusion data segments in the thunder and vision fusion data stream with the road event label information of the thunder and vision fusion data stream to generate aggregated feature vector sets corresponding to each of the thunder and vision fusion data segments.

Step S1222: determining at least one reference potential risk target in the radar-visual fusion data stream based on the aggregated feature vector set.

In this embodiment, the radar-visual fusion data stream received by the server is automatically divided into a plurality of continuous radar-visual fusion data segments. The time length of each radar-visual fusion data segment can be set according to actual needs, for example, one radar-visual fusion data segment is divided every second.

For each radar-visual fusion data segment, the server extracts the tracking features of all road targets and encodes these features into feature vectors. Each radar-visual fusion data segment corresponds to a feature vector set, which contains the feature vectors of all targets in the radar-visual fusion data segment.

The server determines possible road events based on the feature vector set of each radar-vision fusion data segment and assigns corresponding labels and confidence levels to each event. From all possible road event labels, the server selects several labels with the highest confidence levels as the road event label information for the current data segment.

Among them, the server designs a set of aggregation strategies for aggregating the feature vector set of each radar-vision fusion data segment with the road event label information of the radar-vision fusion data segment. The aggregation strategy can include multiple methods such as feature weighting, label embedding, and feature splicing. For each radar-vision fusion data segment, the server first obtains its feature vector set and road event label information. Then, according to the aggregation strategy, the label information is aggregated with each feature vector in the feature vector set in a certain form (such as a label embedding vector). The result of the aggregation is a new feature vector set, namely the aggregated feature vector set, each feature vector in which the original road target tracking features and road event label information are integrated.

The server runs a road object recognition network that is capable of receiving a set of aggregated feature vectors as input and outputting the probability of each object being identified as a potential risk object.

Based on the output of the road target recognition network, the server sets a threshold (such as 0.5) and selects targets with a probability exceeding the threshold as reference potential risk targets. These targets have a high potential risk in the current data segment. The server organizes the selected reference potential risk targets into a list, which contains each target's unique identifier, type, location, speed, driving direction, and associated road event label information. The list will be transmitted to the traffic management department or relevant emergency response agency in real time as part of the road target recognition data.

For example, suppose that within a certain period of time, the radar-visual fusion data stream received by the server is divided into three consecutive data segments. For the first radar-visual fusion data segment, the server extracts a feature vector set and detects two high-confidence road event labels, "pedestrian crossing" and "vehicle emergency stop". The server aggregates the two label information with the feature vector set to generate an aggregated feature vector set.

Next, the server applies the road target recognition network to analyze the aggregated feature vector set and identifies two reference potential risk targets: a vehicle approaching the pedestrian crossing area and a pedestrian crossing the road quickly. The potential risk probabilities of these two targets exceed the set threshold.

Finally, the server compiled the two targets and their related information into a list and transmitted it to the traffic management department in real time. Based on this information, the traffic management department took corresponding traffic control measures in a timely manner, effectively preventing potential traffic accidents.

In a possible implementation, the road target recognition data of the radar-visual fusion data stream is determined by a road target recognition network, and the road target recognition network includes a semantic coding representation unit, a risk target estimation unit, and a risk index evaluation unit.

The semantic coding representation unit is used to obtain the road target tracking features of the radar and vision fusion data stream.

The risk target estimation unit is used to determine at least one reference potential risk target in the radar and visual fusion data stream based on the road target tracking characteristics of the radar and visual fusion data stream.

The risk index evaluation unit is used to determine the potential risk index of each of the reference potential risk targets.

In this embodiment, the server deploys a road target recognition network. The road target recognition network integrates three functional units: semantic coding representation, risk target estimation, and risk index evaluation. It can process the radar and visual fusion data streams from radar and visual sensors in real time, identify potential risk targets on the road, and evaluate their potential risk index.

Among them, the semantic coding representation unit preprocesses the received radar-vision fusion data stream, including data cleaning, denoising, time synchronization and other operations. Subsequently, the feature extraction algorithm in the semantic coding representation unit automatically extracts the tracking features of road targets from each data frame, such as the vehicle's speed, acceleration, position coordinates, driving direction, and pedestrians' walking speed and direction. The extracted features are encoded into vector form, which not only contains numerical information, but also may incorporate semantic information through embedding technology to better reflect the essential attributes and behavior patterns of the target. The encoded feature vector constitutes a high-level abstract representation of the current traffic scene.

The risk target estimation unit receives the road target tracking features (including the encoded feature vector set) from the semantic encoding representation unit. Based on these features, the pattern recognition algorithm in the risk target estimation unit uses deep learning, machine learning, or rule engines to automatically identify road targets that meet potential risk patterns. These patterns may be based on historical accident data, traffic rules, expert experience, and other sources. Through the pattern recognition algorithm, the risk target estimation unit screens out those targets with higher potential risks and marks them as reference potential risk targets. The screening process may involve multiple operations such as confidence calculation, threshold judgment, sorting and interception.

For each reference potential risk target, the risk index assessment unit first retrieves its tracking state path data in the past period of time. These tracking state path data record key information such as the target's motion trajectory, speed change, and direction change. The matching degree calculation algorithm in the risk index assessment unit compares and analyzes the tracking state path data with the global features of the current traffic scene (which may be provided by the semantic encoding representation unit) to calculate the matching degree between the target behavior and the global traffic flow state. The lower the matching degree, the higher the potential risk of the target. Based on the matching degree calculation results, the risk index assessment unit assigns a specific potential risk index to each reference potential risk target. The potential risk index is a quantitative risk assessment result that reflects the possibility of the target causing a traffic accident or traffic congestion in the current traffic scene.

The server integrates the output results of the semantic coding representation unit, the risk target estimation unit and the risk index assessment unit to generate complete road target recognition data, which includes the basic information of all road targets (such as type, location, speed, etc.), the identification of reference potential risk targets and their potential risk index. The server transmits the road target recognition data to the traffic management department or related application platform in real time through the network interface. These data can serve as an important basis for decision support such as traffic monitoring, early warning, and dispatch.

For example, suppose that at a busy intersection, the server receives a radar-visual fusion data stream that contains multiple vehicles on the road and a pedestrian about to cross the road. The semantic coding representation unit first extracts the tracking features of these targets and performs semantic coding. Subsequently, the risk target estimation unit identifies a vehicle that is approaching the zebra crossing quickly and without slowing down as a reference potential risk target. The risk index evaluation unit further analyzes the tracking status path data of the vehicle and calculates that its potential risk index is high. Finally, the server integrates this information into road target recognition data and transmits it to the traffic management department in real time so that they can take timely measures to prevent traffic accidents.

In a possible implementation, the method further includes a step of performing network parameter learning on the road object recognition network, specifically including:

Step S101, using the semantic coding representation unit to obtain a road target tracking feature of a template radar and vision fusion data stream, wherein the road target tracking feature of the template radar and vision fusion data stream reflects the content representation of the template radar and vision fusion data stream.

In this embodiment, during the research and development phase of the intelligent traffic management system, the server is responsible for training and optimizing the road target recognition network. In order to improve the recognition accuracy and robustness of the road target recognition network, the server uses a series of pre-prepared template radar fusion data streams as training data. These template radar fusion data streams contain rich road target tracking information and known risk labels to guide the parameter learning process of the road target recognition network.

In detail, the server loads a template radar-visual fusion data stream from the training data set. The template radar-visual fusion data stream is carefully selected and annotated to reflect the movement of road targets in a specific traffic scenario. Using the semantic coding representation unit, the server preprocesses and extracts features from the template radar-visual fusion data stream. The feature extraction process includes steps such as denoising, time synchronization, target detection and tracking, and finally generates a set of feature vectors containing road target tracking features. These feature vectors not only contain the numerical information of the target (such as position, speed, acceleration), but may also incorporate semantic information to better reflect the essential attributes and behavior patterns of the target.

Step S102, using the risk target estimation unit to determine estimated potential risk target data of the template radar and vision fusion data stream based on the road target tracking characteristics of the template radar and vision fusion data stream, wherein the estimated potential risk target data includes at least one reference potential risk target in the template radar and vision fusion data stream.

In this embodiment, the risk target estimation unit receives the feature vector set from the semantic coding representation unit as input. Based on these features, the pattern recognition algorithm in the risk target estimation unit uses deep learning, machine learning and other technologies to automatically identify road targets that meet potential risk patterns. These patterns may be constructed based on various sources such as historical accident data and traffic rules.

Through the pattern recognition algorithm, the risk target estimation unit screens out those targets with higher potential risks and marks them as estimated potential risk targets. The screening process may involve multiple operations such as confidence calculation, threshold judgment, sorting and interception. Finally, the risk target estimation unit outputs estimated potential risk target data containing at least one reference potential risk target.

Step S103: using the risk index evaluation unit to determine the potential risk index of each of the reference potential risk targets, wherein the potential risk index reflects the abnormal behavior trend parameter of the reference potential risk target in the template radar-visual fusion data stream.

For each reference potential risk target, the risk index evaluation unit retrieves its tracking state path data in the template radar-visual fusion data stream. The matching degree calculation algorithm and risk evaluation model in the risk index evaluation unit are used in combination to first calculate the matching degree between the target behavior and the global traffic flow state, and then evaluate the potential risk index of each target based on the matching degree result. The potential risk index reflects the possibility of the target causing traffic accidents or traffic congestion in the template radar-visual fusion data stream.

Step S104, based on the potential risk index of each of the reference potential risk targets, determine the predicted significant potential risk target data of the template radar-visual fusion data stream, and the predicted significant potential risk target data includes: at least one significant potential risk target determined from the at least one reference potential risk target based on the potential risk index.

Based on the potential risk index, the server further screens out those potential risk targets with the highest risk index and the most significant risk. The screening process may involve sorting and interception operations, such as retaining only the targets with the highest potential risk index as significant potential risk targets.

Finally, the server generates predicted significant potential risk target data containing at least one significant potential risk target, and the predicted significant potential risk target data will be used for subsequent training error calculation and network parameter optimization.

Step S105: determining a first learning cost based on the estimated potential risk target data and the potential risk target prior data, wherein the first learning cost represents a training error of the risk target estimation unit.

Step S106: determining a second learning cost based on the predicted significant potential risk target data and the significant potential risk target prior data, wherein the second learning cost represents a training error of the risk index evaluation unit.

Step S107: performing network parameter learning on the road object recognition network based on the first learning cost and the second learning cost.

In this embodiment, the server loads the potential risk target prior data and the significant potential risk target prior data corresponding to the template radar-visual fusion data stream. These data are pre-labeled and reflect the potential risk targets and significant potential risk targets that actually exist in the template radar-visual fusion data stream.

The server compares the estimated potential risk target data with the potential risk target prior data and calculates the first learning cost (the training error of the risk target estimation unit). Similarly, the server compares the predicted significant potential risk target data with the significant potential risk target prior data and calculates the second learning cost (the training error of the risk index evaluation unit).

Based on the first learning cost and the second learning cost, the server uses optimization algorithms such as gradient descent to adjust the parameters of the road object recognition network. This process aims to minimize the training error and make the network output closer to the actual situation.

The server repeatedly performs the above steps (from feature extraction to parameter learning) and uses multiple different template radar fusion data streams for iterative training. Through multiple iterations, the recognition accuracy and robustness of the network gradually improve.

During the training process, the server monitors the changes in the training error. When the training error stabilizes and no longer decreases significantly, the server determines that the network has converged and the training process ends. The network parameters obtained at this time will be used in actual deployment and online recognition tasks.

In a possible implementation, the estimated potential risk target data includes estimated confidence information corresponding to each radar vision fusion data segment in the template radar vision fusion data stream, and the estimated confidence information includes the estimated confidence of the radar vision fusion data segment relative to multiple potential risk targets. The potential risk target prior data includes prior confidence information corresponding to each radar vision fusion data segment in the template radar vision fusion data stream, and the prior confidence information includes the prior confidence of the radar vision fusion data segment relative to multiple potential risk targets.

Step S105 includes: determining the first learning cost based on estimated confidence information corresponding to each radar-vision fusion data segment in the template radar-vision fusion data stream and prior confidence information corresponding to each radar-vision fusion data segment in the template radar-vision fusion data stream.

In this embodiment, the server first divides the template radar-visual fusion data stream into a plurality of continuous template radar-visual fusion data segments. Each template radar-visual fusion data segment contains the road target tracking information within a certain time range.

For each template radar-vision fusion data segment, the risk target estimation unit uses a pattern recognition algorithm to identify potential risk targets based on the road target tracking characteristics of the data segment, and assigns an estimated confidence to each potential risk target, which reflects the probability that the network believes that the potential risk target exists in the data segment.

The server organizes the estimated confidence information of all data segments into a matrix form. The rows of the matrix correspond to different data segments, and the columns correspond to different potential risk targets. Each element in the matrix represents the estimated confidence of the corresponding data segment for the corresponding potential risk target.

Before training begins, the server has prepared prior data of potential risk targets corresponding to the template radar-vision fusion data stream. These data are obtained through manual annotation or other reliable methods, and reflect the real potential risk targets and their confidence levels in the template radar-vision fusion data stream.

Similar to the estimated confidence information, the server organizes the prior confidence information into a matrix form. The rows and columns of the matrix are the same as the estimated confidence matrix, but the elements in the matrix represent the confidence of the corresponding data segment in the prior knowledge for the corresponding potential risk target.

The server defines a cost function to calculate the first learning cost, which measures the difference between the estimated confidence information and the prior confidence information. Common cost functions include cross entropy loss, mean square error, etc.

Cost calculation: The server traverses each element in the estimated confidence matrix and the prior confidence matrix, and substitutes them into the cost function for calculation. For each data segment and each potential risk target, the server will get a cost value, which reflects the accuracy of the network in identifying the potential risk target on the data segment.

The server aggregates the cost values of all data segments and all potential risk targets to obtain a first learning cost, which is a scalar value that comprehensively reflects the training error of the risk target estimation unit on the entire template radar-vision fusion data stream.

The server can further analyze the source and distribution of the first learning cost. For example, it can identify which data segments or which potential risk targets have large recognition errors, thereby adjusting the network parameters or training strategies in a targeted manner.

For example, assume that the template radar-visual fusion data stream is divided into three data segments, and there are two potential risk targets (such as a vehicle suddenly stopping and a pedestrian crossing). The risk target estimation unit generates an estimated confidence for each data segment for each potential risk target. At the same time, the server also obtains the corresponding prior confidence information.

The server organizes this information into a matrix form and applies a defined cost function (such as cross entropy loss) to calculate the cost value of each data segment and each potential risk target. Finally, the server adds up all the cost values to get the first learning cost. By analyzing this first learning cost, the server can evaluate the training effect of the risk target estimation unit and adjust the parameters of the network accordingly to further reduce the training error.

In a possible implementation manner, the prior data of significant potential risk targets includes at least one prior significant potential risk target in the template radar-visual fusion data stream.

Step S106 includes: determining the second learning cost based on the potential risk index corresponding to each of the reference potential risk targets in the template radar-visual fusion data stream, and the priori significant potential risk target in each of the reference potential risk targets.

Before training begins, the server has prepared prior data of significant potential risk targets. These prior data of significant potential risk targets are obtained through manual labeling or other reliable methods, and identify at least one target in the template radar-vision fusion data stream that is confirmed as a significant potential risk. These targets usually have extremely high potential risk indexes and pose a serious threat to traffic safety.

The prior data of significant potential risk targets are stored in the database of the server so that they can be called at any time during the training process. These data may exist in the form of target identifiers, locations, timestamps, potential risk indexes, etc.

The risk index evaluation unit receives the estimated potential risk target data from the risk target estimation unit, and calculates the potential risk index for each reference potential risk target based on the estimated potential risk target data. The potential risk index reflects the probability of the target causing a traffic accident or traffic congestion in the template radar-visual fusion data stream. Based on the potential risk index, the server further selects the potential risk targets with the highest risk index and the most significance, which constitute the predicted significance potential risk target data.

The server matches the predicted significant potential risk target data with the significant potential risk target prior data. The matching process may involve comparing target identifiers, locations, timestamps, and other information to determine which predicted targets are significant potential risk targets confirmed in prior knowledge.

The server defines a cost function to calculate the second learning cost, which measures the difference between the predicted significant potential risk target and the prior significant potential risk target. The design of the cost function may take into account multiple factors, such as the number of predicted targets, the accuracy of the predicted targets (i.e., the consistency of the predicted targets with the prior targets), the deviation of the potential risk index of the predicted targets from the potential risk index of the prior targets, etc.

The server calculates the second learning cost according to the cost function. The calculation process may involve comparing the predicted target with the prior target one by one and accumulating the cost value of each comparison result. The final second learning cost is a scalar value, which comprehensively reflects the training error of the risk index assessment unit in identifying significant potential risk targets.

Error analysis: The server can also further analyze the source and distribution of the second learning cost. For example, it can identify which predicted targets are incorrectly marked as significant potential risk targets, which prior significant potential risk targets are omitted, etc. These analysis results help the server adjust the parameters or training strategies of the risk index evaluation unit to further improve its recognition accuracy.

For example, suppose the template radar-visual fusion data stream contains tracking information of multiple vehicles and pedestrians, and prior knowledge confirms that one of the vehicles (vehicle A) has an extremely high potential risk index at a certain moment and is a significant potential risk target. The risk index evaluation unit calculates the potential risk index for multiple reference potential risk targets based on the input data and selects the predicted significant potential risk target data.

The server matches the predicted data with the prior data and finds that the predicted data contains vehicle A as a significant potential risk target. However, the predicted data may also contain other significant potential risk targets that are not confirmed in the prior knowledge (such as false positive cases). The server calculates the second learning cost based on the cost function, which reflects the degree of difference between the predicted data and the prior data. By analyzing this second learning cost, the server can evaluate the training effect of the risk index evaluation unit and adjust the network parameters or training strategy accordingly to improve performance.

In a possible implementation, the road object recognition network further includes a fully connected output unit, and the method further includes:

Step A110, using the fully connected output unit to determine the road event label estimation data of the template radar-vision fusion data stream based on the road target tracking characteristics of the template radar-vision fusion data stream, the road event label estimation data of the template radar-vision fusion data stream includes estimated confidences corresponding to m road event labels respectively, and the estimated confidence corresponding to the p-th road event label among the m road event labels, reflecting the estimated confidence that the template radar-vision fusion data stream matches the p-th road event label, and p is not greater than m.

Step A120, based on the road event label estimation data of the template radar-vision fusion data stream, determine the road event label information of the template radar-vision fusion data stream, the road event label information of the template radar-vision fusion data stream reflects the label knowledge point information corresponding to the road event of the template radar-vision fusion data stream, the risk target estimation unit is used to determine the estimated potential risk target data of the template radar-vision fusion data stream based on the road target tracking characteristics of the template radar-vision fusion data stream and the road event label information of the template radar-vision fusion data stream.

Step A130, determining a third learning cost based on the road event label estimation data and the road event label prior data, wherein the third learning cost represents a training error of the fully connected output unit.

Step S107 includes: performing network parameter learning on the road object recognition network based on the first learning cost, the second learning cost and the third learning cost.

In this embodiment, the fully connected output unit receives the road target tracking features from the semantic encoding representation unit as input. These road target tracking features have been preprocessed and encoded, and contain detailed information of all road targets in the template radar-visual fusion data stream. The fully connected layer in the fully connected output unit performs weighted summation and nonlinear transformation on the input features, and outputs the estimated confidences corresponding to the m road event labels, which reflect the probability that the template radar-visual fusion data stream matches each road event label. For example, if m=5, the unit will output 5 confidence values, corresponding to event labels such as "vehicle emergency stop", "pedestrian crossing", "traffic accident", "traffic congestion" and "normal driving".

The server organizes the m road event labels and their corresponding estimated confidences into road event label estimation data, which will be used for subsequent road event label information determination and network parameter learning.

Next, the server has a label knowledge base, which contains detailed descriptions of each road event label, historical cases, countermeasures and other knowledge point information. For labels with higher confidence in the road event label estimation data, the server will query the knowledge base to obtain richer label knowledge point information. The server combines the queried label knowledge point information with the estimated confidence to generate road event label information for the template radar-visual fusion data stream, which not only reflects the types of road events that may occur in the data stream, but also provides detailed background and countermeasures for these events.

The risk target estimation unit now receives not only the road target tracking features as input, but also the road event label information, which together constitute the comprehensive feature set required to estimate potential risk targets. Based on the comprehensive feature set, the risk target estimation unit uses a pattern recognition algorithm to identify potential risk targets in the template radar-visual fusion data stream. The identification process of these targets may be significantly affected by the road event label information, because some events (such as traffic accident scenes) themselves will increase the potential risk of surrounding targets.

The server loads the road event label prior data corresponding to the template radar-visual fusion data stream. These road event label prior data are pre-labeled and reflect the road events that actually occurred in the template radar-visual fusion data stream and their confidence. The server compares the road event label estimation data with the prior data and calculates the third learning cost, which reflects the training error of the fully connected output unit in identifying the road event label. The cost function may take into account multiple factors, such as the accuracy of the label, the deviation of the confidence, etc.

The server summarizes the first learning cost (training error of the risk target estimation unit), the second learning cost (training error of the risk index evaluation unit), and the third learning cost (training error of the fully connected output unit), which together reflect the overall performance of the road target recognition network in the current training stage.

Based on the aggregated learning cost, the server uses gradient descent or other optimization algorithms to adjust the parameters of the network. The adjustment process aims to minimize the learning cost and make the network output closer to the actual situation. The server repeats the above steps (from feature extraction to parameter learning) and uses multiple different template radar fusion data streams for iterative training. Through multiple iterations, the recognition accuracy and robustness of the network gradually improve. During the training process, the server monitors the changes in the learning cost. When the learning cost tends to stabilize and no longer decreases significantly, the server determines that the network has converged and the training process ends. The network parameters obtained at this time will be used in actual deployment and online recognition tasks.

In a possible implementation, the road event label prior data includes the prior confidences corresponding to the m road event labels respectively, and the prior confidence corresponding to the p-th road event label reflects the prior confidence that the template radar-visual fusion data stream matches the p-th road event label.

Step A130 includes: determining the third learning cost based on the estimated confidences respectively corresponding to the m road event labels and the prior confidences respectively corresponding to the m road event labels.

In this embodiment, before the training begins, the server generates road event label prior data through manual annotation or other reliable methods. These road event label prior data contain each possible road event label (a total of m) in the template radar-visual fusion data stream and its corresponding prior confidence. The prior confidence reflects the natural probability or expected probability that the data stream matches each event label in the absence of any additional information. The server stores these prior data in the database so that they can be called at any time during the training process. The prior data may exist in the form of label identifiers, event descriptions, prior confidence, etc., forming a structured data set.

The fully connected output unit receives the road object tracking features from the semantic encoding representation unit as input, and predicts the road event labels and their estimated confidences that the template data stream may match based on these features. The unit outputs a list of m event labels and their corresponding estimated confidences.

The server matches the estimated road event label data with the road event label prior data. The matching process involves aligning each event label in the estimated data with its corresponding label in the prior data so that their confidences can be compared later. In order to quantify the difference between the estimated confidence and the prior confidence, the server defines a cost function that may take into account multiple factors, such as the absolute difference in confidence, relative error, whether the label is correctly identified, etc. Common cost functions include cross entropy loss, mean square error, etc. The specific choice depends on the training objective and optimization strategy.

The server traverses the matched data pairs (i.e., each event label and its corresponding estimated confidence and prior confidence) and substitutes them into the cost function for calculation. For each event label, the server obtains a cost value that reflects the prediction accuracy of the fully connected output unit on that label. Finally, the server sums up the cost values of all event labels to obtain the third learning cost, which is a scalar value that comprehensively reflects the average error or overall performance of the fully connected output unit in identifying road event labels on the entire template data stream.

For example, suppose there are three possible matching road event labels in the template radar-vision fusion data stream: "vehicle emergency stop", "pedestrian crossing" and "normal driving" (i.e. m=3). The server has prepared the prior confidences corresponding to these three labels and stored them in the database.

The fully connected output unit predicts the estimated confidence of these three labels based on the input road target tracking features. For example, for the "vehicle emergency stop" label, the prior confidence can be 0.05 (indicating that this event is not very common in general), while the estimated confidence can be 0.8 (indicating that the unit believes that the current data stream is likely to match this label).

The server matches the estimated confidence with the prior confidence and applies a defined cost function (e.g., cross entropy loss) to calculate the cost. For the "vehicle emergency stop" label, since the estimated confidence is much higher than the prior confidence, the cost may be relatively large, reflecting the difference between the unit's prediction on this label and the prior knowledge.

Similarly, the server calculates the cost values for the other two labels and sums them up to get the third learning cost. By analyzing this cost value, the server can evaluate the performance of the fully connected output unit in identifying road event labels and adjust the network parameters or training strategy accordingly to improve performance.

The embodiment of the present application provides a road target recognition system based on radar and vision fusion for implementing the above-mentioned road target recognition method based on radar and vision fusion, including a processor, a machine-readable storage medium, a bus, and a communication unit.

In a possible design, the road target recognition system based on thunder and vision fusion can be a single server or a server group. The server group can be centralized or distributed (for example, the road target recognition system based on thunder and vision fusion can be a distributed system). In some embodiments, the road target recognition system based on thunder and vision fusion can be local or remote. For example, the road target recognition system based on thunder and vision fusion can access information and/or data stored in a machine-readable storage medium via a network. For another example, the road target recognition system based on thunder and vision fusion can be directly connected to a machine-readable storage medium to access stored information and/or data. In some embodiments, the road target recognition system based on thunder and vision fusion can be implemented on a road target recognition system based on thunder and vision fusion. As an example only, the road target recognition system based on thunder and vision fusion can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc. or any combination thereof.

The machine-readable storage medium may store data and/or instructions. In some embodiments, the machine-readable storage medium may store data acquired from an external terminal. In some embodiments, the machine-readable storage medium may store data and/or instructions used by a road target recognition system based on radar-visual fusion to execute or use to complete the exemplary method described in this application.

In the specific implementation process, one or more processors execute computer executable instructions stored in a machine-readable storage medium, so that the processor can execute the road target recognition method based on radar-visual fusion in the above method embodiment. The processor, the machine-readable storage medium and the communication unit are connected through a bus, and the processor can be used to control the sending and receiving actions of the communication unit.

The specific implementation process of the processor can refer to the various method embodiments executed by the above-mentioned road target recognition system based on radar and vision fusion. The implementation principles and technical effects are similar and will not be repeated in this embodiment.

In addition, an embodiment of the present application further provides a readable storage medium, in which computer executable instructions are preset. When a processor executes the computer executable instructions, the above-mentioned road target recognition method based on radar-visual fusion is implemented.

It should be noted that in order to simplify the description disclosed in this application and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, multiple features are sometimes combined into one embodiment, drawings, or descriptions thereof. Similarly, it should be noted that in order to simplify the description disclosed in this application and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, multiple features are sometimes combined into one embodiment, drawings, or descriptions thereof.

Claims (5)

1. A road target recognition method based on radar and visual fusion, characterized in that the method comprises: Acquire a road target tracking feature of a radar-vision fusion data stream, wherein the road target tracking feature of the radar-vision fusion data stream reflects a content representation of the radar-vision fusion data stream; Based on the road target tracking characteristics of the radar and visual fusion data stream, determining at least one reference potential risk target in the radar and visual fusion data stream; Acquire the tracking state path data of each of the reference potential risk targets, and determine the potential risk index of each of the reference potential risk targets based on the tracking state path data of each of the reference potential risk targets and the road target tracking characteristics of the radar-visual fusion data stream, wherein the potential risk index reflects the abnormal behavior trend parameter of the reference potential risk target in the radar-visual fusion data stream; Determining the road target identification data of the radar-visual fusion data stream based on the potential risk index of each of the reference potential risk targets; The determining of at least one reference potential risk target in the radar and visual fusion data stream based on the road target tracking feature of the radar and visual fusion data stream comprises: Based on the road target tracking characteristics of the thunder and vision fusion data stream, determining the road event label information of the thunder and vision fusion data stream, wherein the road event label information of the thunder and vision fusion data stream reflects the label knowledge point information corresponding to the road event of the thunder and vision fusion data stream; Determine at least one reference potential risk target in the radar and vision fusion data stream based on the road target tracking characteristics of the radar and vision fusion data stream and the road event label information of the radar and vision fusion data stream; Wherein, determining the road event label information of the radar and visual fusion data stream based on the road target tracking feature of the radar and visual fusion data stream includes: Based on the road target tracking characteristics of the radar-visual fusion data stream, the confidences corresponding to the z road event labels are determined, wherein the confidence corresponding to the x-th road event label among the z road event labels reflects that the probability x that the radar-visual fusion data stream matches the x-th road event label is not greater than z; Based on the confidences respectively corresponding to the z road event labels, y road event labels with the largest confidences are selected from the z road event labels to generate y target road event labels; Based on the confidences respectively corresponding to the y target road event labels, the label knowledge features respectively corresponding to the y target road event labels are fused and calculated to generate a global label knowledge feature, and the road event label information of the radar-visual fusion data stream includes the global label knowledge feature; The road target tracking feature of the radar-visual fusion data stream includes a set of feature vectors corresponding to a plurality of radar-visual fusion data segments in the radar-visual fusion data stream; The determining at least one reference potential risk target in the radar and vision fusion data stream based on the road target tracking feature of the radar and vision fusion data stream and the road event label information of the radar and vision fusion data stream comprises: Aggregating the feature vector sets corresponding to each of the thunder and vision fusion data segments in the thunder and vision fusion data stream with the road event label information of the thunder and vision fusion data stream, to generate aggregated feature vector sets corresponding to each of the thunder and vision fusion data segments; Based on the aggregated feature vector set, determining at least one reference potential risk target in the radar-visual fusion data stream; The road target recognition data of the radar-visual fusion data stream is determined by a road target recognition network, which includes a semantic coding representation unit, a risk target estimation unit and a risk index evaluation unit; The semantic coding representation unit is used to obtain the road target tracking features of the radar and vision fusion data stream; The risk target estimation unit is used to determine at least one reference potential risk target in the radar and visual fusion data stream based on the road target tracking characteristics of the radar and visual fusion data stream; The risk index evaluation unit is used to determine the potential risk index of each of the reference potential risk targets; The method further comprises the step of performing network parameter learning on the road object recognition network, specifically comprising: The semantic coding representation unit is used to obtain a road target tracking feature of a template radar and vision fusion data stream, wherein the road target tracking feature of the template radar and vision fusion data stream reflects a content representation of the template radar and vision fusion data stream; Determine estimated potential risk target data of the template radar-visual fusion data stream by using the risk target estimation unit based on the road target tracking characteristics of the template radar-visual fusion data stream, wherein the estimated potential risk target data includes at least one reference potential risk target in the template radar-visual fusion data stream; Determine the potential risk index of each of the reference potential risk targets by using the risk index evaluation unit, wherein the potential risk index reflects the abnormal behavior trend parameter of the reference potential risk target in the template radar-visual fusion data stream; Based on the potential risk index of each of the reference potential risk targets, predicting significant potential risk target data of the template radar-visual fusion data stream, the predicted significant potential risk target data comprising: at least one significant potential risk target determined from the at least one reference potential risk target based on the potential risk index; Determining a first learning cost based on the estimated potential risk target data and the potential risk target prior data, wherein the first learning cost represents a training error of the risk target estimation unit; Determining a second learning cost based on the predicted significant potential risk target data and the significant potential risk target prior data, wherein the second learning cost represents a training error of the risk index evaluation unit; Based on the first learning cost and the second learning cost, performing network parameter learning on the road object recognition network; The estimated potential risk target data includes estimated confidence information corresponding to each radar vision fusion data segment in the template radar vision fusion data stream, and the estimated confidence information includes estimated confidence of the radar vision fusion data segment relative to multiple potential risk targets; the potential risk target prior data includes prior confidence information corresponding to each radar vision fusion data segment in the template radar vision fusion data stream, and the prior confidence information includes prior confidence of the radar vision fusion data segment relative to multiple potential risk targets; The determining of a first learning cost based on the estimated potential risk target data and the potential risk target prior data includes: Determining the first learning cost based on the estimated confidence information corresponding to each radar-vision fusion data segment in the template radar-vision fusion data stream and the priori confidence information corresponding to each radar-vision fusion data segment in the template radar-vision fusion data stream; Wherein, the prior data of significant potential risk targets includes at least one prior significant potential risk target in the template radar-visual fusion data stream; The determining of the second learning cost based on the predicted significant potential risk target data and the significant potential risk target prior data includes: Determining the second learning cost based on the potential risk indexes respectively corresponding to the reference potential risk targets in the template radar-visual fusion data stream and the priori significant potential risk targets in the reference potential risk targets; Wherein, the road object recognition network further includes a fully connected output unit, and the method further includes: Determine the road event label estimation data of the template radar-vision fusion data stream based on the road target tracking feature of the template radar-vision fusion data stream by using the fully connected output unit, wherein the road event label estimation data of the template radar-vision fusion data stream includes estimated confidences corresponding to m road event labels respectively, and the estimated confidence corresponding to the p-th road event label among the m road event labels, reflecting the estimated confidence that the template radar-vision fusion data stream matches the p-th road event label, and p is not greater than m; Based on the road event label estimation data of the template radar-vision fusion data stream, the road event label information of the template radar-vision fusion data stream is determined, the road event label information of the template radar-vision fusion data stream reflects the label knowledge point information corresponding to the road event of the template radar-vision fusion data stream, and the risk target estimation unit is used to determine the estimated potential risk target data of the template radar-vision fusion data stream based on the road target tracking characteristics of the template radar-vision fusion data stream and the road event label information of the template radar-vision fusion data stream; Determining a third learning cost based on the road event label estimation data and the road event label prior data, wherein the third learning cost represents a training error of the fully connected output unit; The performing network parameter learning on the road object recognition network based on the first learning cost and the second learning cost includes: Based on the first learning cost, the second learning cost and the third learning cost, performing network parameter learning on the road object recognition network; The road event label prior data includes the prior confidences corresponding to the m road event labels respectively, the prior confidence corresponding to the p-th road event label, and reflects the prior confidence that the template radar-visual fusion data stream matches the p-th road event label; The determining of the third learning cost based on the road event label estimation data and the road event label prior data includes: The third learning cost is determined based on the estimated confidences respectively corresponding to the m road event labels and the priori confidences respectively corresponding to the m road event labels. 2. The road target recognition method based on radar and visual fusion according to claim 1, characterized in that the road target tracking features of the radar and visual fusion data stream include a first feature vector set, and the first feature vector set reflects the global content representation of the radar and visual fusion data stream; The step of determining the potential risk index of each reference potential risk target based on the tracking state path data of each reference potential risk target and the road target tracking characteristics of the radar-visual fusion data stream comprises: Based on the matching degree between the tracking state path data of each reference potential risk target and the first feature vector set, a potential risk index of each reference potential risk target is determined. 3. The method for road target identification based on radar and visual fusion according to claim 1, characterized in that the road target identification data of the radar and visual fusion data stream is determined based on the potential risk index of each reference potential risk target, comprising: Outputting the reference potential risk target whose potential risk index meets the first setting requirement as the potential risk target included in the radar-visual fusion data stream; Among them, the first setting requirement includes that the potential risk index is not less than a first threshold value, or that each of the reference potential risk targets is arranged in descending order of the potential risk index and is located in the first i order of the arrangement result, where i is a set positive integer. 4. The method for road target identification based on radar and visual fusion according to claim 1, characterized in that the road target identification data of the radar and visual fusion data stream is determined based on the potential risk index of each reference potential risk target, comprising: Outputting the reference potential risk target whose potential risk index meets the second setting requirement as a significant potential risk target included in the radar-visual fusion data stream; Among them, the second setting requirement includes: the potential risk index is not less than a second threshold value, or each of the reference potential risk targets is arranged in descending order according to the potential risk index and is located in the first j order of the arrangement result, where j is a set positive integer. 5. A road target recognition system based on radar and vision fusion, characterized in that the road target recognition system based on radar and vision fusion includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the road target recognition method based on radar and vision fusion described in any one of claims 1 to 4.