CN117333627A - Reconstruction and complement method, system and storage medium for automatic driving scene - Google Patents

Abstract

The invention provides a method, a system and a storage medium for reconstructing and complementing an automatic driving scene. The method comprises the following steps: acquiring a stereoscopic camera image of an automatic driving scene, and generating a depth map and a semantic map; optimizing a depth map by using the semantic map; establishing a local surface element model by utilizing the optimized depth map, eliminating conflict information in the local surface element model, and fusing the local surface element model and the global surface element model to obtain a fused global surface element model; acquiring a scene image with a new view angle from the global surface element model after the construction of the whole scene is completed; because of the estimation error in the construction process of the face element model and the removal of the dynamic target object, various holes/masks can appear in the new view angle scene rendering image, the context information of the reference image is matched to obtain context associated information of the new view angle scene rendering image, and the context associated information comprises a similarity graph and an associated image; and complementing the new view angle scene rendering image by using the countermeasure generation network model.

Description

A method, system and storage medium for reconstruction and completion of autonomous driving scenes

Technical field

The present invention relates to the technical field of image scene generation, and specifically relates to a method, system and storage medium for reconstructing and completing automatic driving scenes.

Background technique

Currently, the development of autonomous driving-related applications requires extensive verification and testing of autonomous vehicles to ensure the safety of their implementation. One of its solutions is to realistically reproduce a large number of complex and diverse traffic scenes from different trajectories and angles. Therefore, the authenticity and consistency of the constructed scene flow are crucial for autonomous driving system testing. Among them, the aforementioned scenes can be created through game engines or high-fidelity computer graphics (CG) models, such as Intel's CARLA (an open source simulator for autonomous driving research), Microsoft's AirSim (a cross-platform simulator built on Unreal Engine) Simulators for drones and other autonomous mobile devices), Google's CarCraft (a software for testing self-driving cars in a virtual reconstructed city), etc. can all achieve excellent rendering effects, but their synthetic images still lack the richness of real-world images. properties, leading to performance degradation. Another option is a data-driven approach that relies on sensory data (such as data acquired by cameras and lidar) to reconstruct the aforementioned traffic scene. The reconstructed traffic scene can retain rich information about semantics, scene lighting and appearance. Generally, the data-driven scene generation process consists of three parts: first, denoising and refining the aforementioned sensory data; then reconstructing the 3D model using appropriate geometric agents to generate ordered static scenes; and finally using image enhancement techniques to improve Image quality and consistency. However, in order to achieve effective static scenes, new perspective image synthesis needs to overcome the following challenges:

(1) Construct a scene geometric model. Cameras can capture higher measurement resolution and more semantic information than lidar, but require more precise depth estimates for model reconstruction. At the same time, the semantic information of the image can help improve the estimated depth. How to effectively utilize stereo cameras to achieve high-precision, high-resolution surfing models remains a difficult problem;

(2) Completion of missing parts in generated images. After constructing a three-dimensional background model, various irregular holes will appear in the generated new view image due to estimation errors and removal of dynamic targets. Once the 3D background model is established, these occluded areas cannot be seen from any angle. How to fill these holes that cannot be ignored with certain content and generate reasonably realistic images is a major issue;

(3) Spatiotemporal consistency between the generated new view images. For highly structured new view images with small holes or pores, its neighboring pixels can be exploited to infer the missing parts, but this approach does not work well for complex and diverse traffic scenes. Furthermore, most image completion algorithms inevitably lead to temporal artifacts and jitter. How to generate a spatiotemporally consistent new view image sequence from a new perspective is of great significance for the verification and testing of functional modules in autonomous driving systems.

Moving cameras can generate dense 3D point clouds through multi-view stereo (MVS) technology, but this requires acquiring multiple images around the same area from different angles, while moving objects severely degrades their performance. Compared with MVS, stereo matching methods can be used for 3D reconstruction of outdoor environments from high-resolution images, but a major problem is the lack of ground-truth data for high-performance models. In order to improve the quality of 3D reconstruction, researchers have also developed some methods using semantic information, which usually involve high computational complexity and joint learning architectures, and still contain many outliers.

Under dense depth conditions, the construction error of the surface element model (i.e., the above-mentioned three-dimensional background model) is small, and the parameters can be flexibly adjusted. However, the rendering performance of new view images relies on the quality of the 3D point cloud, which often cannot cover the entire scene, so advanced image completion technology is required. Recently, methods based on Generative Adversarial Network (GAN) have been developed, using self-learning encoder-decoder models, extended convolutions that form multi-scale contextual information, and global and local discriminators to complete the image. However, these methods usually cannot handle images of complex scenes and large holes. Combined with patch-based learning methods, they can borrow similar features from surrounding visible areas through patch-based attention mechanisms. However, these techniques attempt to find similar patches within or between images and are not suitable for generating content that is not in the visible area. Furthermore, they do each image individually without considering the spatial consistency between the above new view image sequences. However, the Conditional Normalization Layer (CNL) can utilize additional images to achieve global spatial encoding and spatial variation adjustment.

Therefore, it is necessary to improve the above-mentioned prior art.

Contents of the invention

The main technical problem solved by the present invention is to provide a method, system and storage medium for reconstructing and completing autonomous driving scenes, so as to use the optimized depth map to establish an initial local surface element model, and to Conflict information in the element model that conflicts with the initial global scene element model is eliminated, and the initial global scene element model is fused using the local plane element model after eliminating the conflict information to obtain the fused global scene element model. Meta model; after obtaining the global element model, the new perspective scene rendering image is rendered through an adaptive method, and finally the adversarial generative network model is used to generate a high-quality complete new perspective scene image sequence with consistent spatial structure.

According to a first aspect, an embodiment provides a reconstruction and completion method of an autonomous driving scene. The method includes:

Obtain a stereo camera image of the automatic driving scene, and generate a depth map and a semantic map of the stereo camera image; wherein the stereo camera image is obtained by shooting the automatic driving scene by a stereo camera; the depth The graph is used to represent the depth information of the stereo camera image, and the semantic map is used to represent the semantic information of the stereo camera image; the semantic map is used to optimize the depth map to obtain an optimized depth map;

Using the optimized depth map to establish a local block model of the current frame of the stereo camera image, the local block model of the current frame and the global block model corresponding to all image frames before the current frame Conflicting conflict information is eliminated, and the local surface element model of the current frame after eliminating the conflict information is used to fuse with the global scene element model until the global scene corresponding to all image frames of the autonomous driving scene is The metamodel construction is completed;

Perform adaptive rendering on the global scene element model corresponding to all image frames of the autonomous driving scene to obtain the adaptively rendered global scene element model. From the adaptively rendered global scene element model Obtain a new perspective scene image sequence corresponding to the autonomous driving scene in the model; wherein the new perspective scene image sequence includes a plurality of new perspective scene rendering images;

Obtain the rendered image pyramid of the new perspective scene rendering image, the reference image pyramid of the reference image corresponding to the new perspective scene rendering image, and the mask image pyramid of the mask image corresponding to the new perspective scene rendering image, and The rendered image pyramid, the reference image pyramid and the mask image pyramid input a trained adversarial generative network model to use the adversarial generative network model to supplement the areas with missing information in the new perspective scene rendering image. Completely generate a complete new perspective scene image sequence;

Wherein, the complete new perspective scene image sequence includes multiple complete new perspective scene images; the reference image is the stereo camera image from an adjacent perspective of the new perspective scene rendering image, and the mask image is based on The new perspective scene is obtained by rendering the area with missing information in the image.

According to the second aspect, an embodiment provides a reconstruction and completion system for autonomous driving scenes. The reconstruction and completion system includes:

a stereo camera configured to acquire a stereo camera image of the autonomous driving scene;

a semantic depth generation module configured to generate a depth map and a semantic map of the stereo camera image;

A semantic depth enhancement module configured to optimize the depth map using the semantic map to obtain an optimized depth map;

A surface element model building module configured to use the optimized depth map to establish a local surface element model of the current frame of the stereo camera image, and compare the local surface element model of the current frame with the previous one of the current frame. The conflict information that conflicts with the global scene element model corresponding to all image frames is eliminated, and the local segment model of the current frame after eliminating the conflict information is used to fuse with the global scene element model until it is integrated with the automatic driving The construction of the global scene element model corresponding to all image frames of the scene is completed; the global scene element model corresponding to all the image frames of the autonomous driving scene is adaptively rendered to obtain the adaptively rendered global scene element model. , obtain a new perspective scene image sequence corresponding to the autonomous driving scene from the adaptively rendered global element model; wherein the new perspective scene image sequence includes a plurality of new perspective scene rendering images;

An adversarial network completion module configured to obtain a rendered image pyramid of the new perspective scene rendering image, a reference image pyramid of a reference image corresponding to the new perspective scene rendering image, and a mask corresponding to the new perspective scene rendering image. Mask image pyramid of the code image, input the rendered image pyramid, the reference image pyramid and the mask image pyramid into a trained adversarial generative network model to use the adversarial generative network model to analyze the new perspective scene The areas with missing information in the rendered image are completed to generate a complete new perspective scene image sequence;

Wherein, the complete new perspective scene image sequence includes multiple complete new perspective scene images; the reference image is the stereo camera image from an adjacent perspective of the new perspective scene rendering image, and the mask image is based on The new perspective scene is obtained by rendering the area with missing information in the image.

According to a third aspect, an embodiment provides a computer-readable storage medium. The computer-readable storage medium includes the program. The program can be executed by the processor to implement the method as described in any embodiment herein.

The beneficial effects of this application are:

The reconstruction and completion method, system and storage medium of this application generate the depth map and semantic map of the stereo camera image by acquiring the stereo camera image of the autonomous driving scene; wherein the stereo camera image is obtained by shooting the autonomous driving scene with a stereo camera. Obtained; the depth map is used to characterize the depth information of the stereo camera image, and the semantic map is used to characterize the semantic information of the stereo camera image; the semantic map is used to optimize the depth map to obtain an optimized depth map; the optimized depth map is used Establish a local block model of the current frame of the stereo camera image, and eliminate the conflict information that conflicts with the local block model of the current frame and the global block model corresponding to all image frames before the current frame, and use the conflict information after eliminating the conflict information The local segment model of the current frame is merged with the global segment model until the global segment model corresponding to all image frames of the autonomous driving scene is completed; the global segment model corresponding to all image frames of the autonomous driving scene is automatically Adapt to rendering to obtain an adaptively rendered global scene element model, and obtain a new perspective scene image sequence corresponding to the autonomous driving scene from the adaptively rendered global scene element model; wherein the new perspective scene image sequence includes multiple new perspectives Scene rendering image; obtain the rendering image pyramid of the new perspective scene rendering image, the reference image pyramid of the reference image corresponding to the new perspective scene rendering image, and the mask image pyramid of the mask image corresponding to the new perspective scene rendering image, and will render the image The pyramid, reference image pyramid and mask image pyramid input the trained adversarial generative network model to use the adversarial generative network model to complete the missing information areas in the new perspective scene rendering image to generate a complete new perspective scene image sequence; where, The complete new perspective scene image sequence includes multiple complete new perspective scene images; the reference image is a stereo camera image from an adjacent perspective of the new perspective scene rendering image, and the mask image is obtained based on the missing information area in the new perspective scene rendering image. of.

Description of drawings

Figure 1 is a schematic flow chart of a reconstruction and completion method according to an embodiment;

Figure 2 is a schematic flowchart of an embodiment of using an adversarial generative network model to complete areas with missing information in a new perspective scene rendering image to generate a complete new perspective scene image sequence;

Figure 3 is a schematic diagram of the data processing flow inside the CSD residual module according to an embodiment;

Figure 4 is a schematic diagram of the data processing flow of the CSD sub-module according to an embodiment;

Figure 5 is a schematic diagram of obtaining contextual information related to the image result of the corresponding layer in the rendered image pyramid according to an embodiment;

Figure 6 is a schematic diagram of similarity values of a rendering sub-block and multiple reference mask sub-blocks in a search domain corresponding to the rendering sub-block according to an embodiment;

Figure 7 is a schematic module diagram of a generator of an adversarial generative network model according to an embodiment;

Figure 8 is a schematic module diagram of an autonomous driving scene reconstruction and completion system according to an embodiment.

Detailed ways

The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments use associated similar element numbers. In the following embodiments, many details are described in order to make the present application better understood. However, those skilled in the art can readily recognize that some of the features may be omitted in different situations, or may be replaced by other elements, materials, and methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core part of the present application being overwhelmed by excessive descriptions. For those skilled in the art, it is difficult to describe these in detail. The relevant operations are not necessary, and they can fully understand the relevant operations based on the descriptions in the instructions and general technical knowledge in the field.

Additionally, the features, operations, or characteristics described in the specification may be combined in any suitable manner to form various embodiments. At the same time, each step or action in the method description can also be sequentially exchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the description and drawings are only for clearly describing a certain embodiment, and do not imply a necessary sequence, unless otherwise stated that a certain sequence must be followed.

The serial numbers assigned to components in this article, such as "first", "second", etc., are only used to distinguish the described objects and do not have any sequential or technical meaning. The terms "connection" and "connection" mentioned in this application include direct and indirect connections (connections) unless otherwise specified.

The technical concept of the reconstruction and completion method of autonomous driving scenes and its system provided by this application is: first, use the stereo camera images of the autonomous driving scene acquired by the stereo camera to construct a three-dimensional geometric model for representing large-scale traffic scenes (such as the global element model below); among them, this is because the above-mentioned stereo camera images can be processed through high-precision learning methods (such as semantic segmentation, depth prediction) to provide dense depth information for the construction of the aforementioned three-dimensional geometric model ( (such as the depth map below) and semantic information (such as the semantic map below); then, use the aforementioned semantic segmentation results (such as the semantic map below) to further effectively optimize the aforementioned depth information, and then use the optimized aforementioned depth information to Achieve high-quality reconstruction of the three-dimensional geometric model; then, for the areas with missing information in the new-view scene rendering image in the new-view scene image sequence obtained from the reconstructed three-dimensional geometric model, search and search from the collected reference images and The aforementioned new perspective scene rendering image has similar sub-blocks or areas, and multiple new perspective scene rendering images of different scales are matched with the above reference image in terms of contextual content, and the matching correlation results are converted into subsequent feature spaces. The normalized modulation affine parameters are trained to generate the texture of the areas with missing information in the above-mentioned new perspective scene rendering image, thereby ensuring the consistency in content and structure of the final generated complete new perspective scene image sequence. Among them, for the construction of the above three-dimensional geometric model, the required input includes stereo camera images of autonomous driving scenes, and the corresponding depth maps and semantic maps are generated using the stereo camera images. The semantic map is used to optimize the boundary anomalies in the depth map and remove dynamic targets. The optimized depth map is used to establish the initial local surface element model, and the conflict between the initial local surface element model and the initial global surface element model is analyzed. Elimination is performed, and the global scene element model increases with the increase of the fusion strategy (that is, the process of fusing the initial global scene element model). After obtaining the global element model, the new perspective scene rendering image is rendered through an adaptive method, and finally the adversarial generative network model is used to generate a high-quality complete new perspective scene image sequence with consistent spatial structure.

The technical solutions of the present application will be described in detail below with reference to examples.

This application provides a method for reconstructing and completing autonomous driving scenes. Please refer to Figure 1. The reconstruction and completion methods include:

Step S100: Obtain the stereo camera image of the autonomous driving scene, and generate the depth map and semantic map of the stereo camera image; where the stereo camera image is obtained by shooting the autonomous driving scene with a stereo camera; the depth map is used to characterize the stereo camera image. The depth information, the semantic map is used to characterize the semantic information of the stereo camera image; the semantic map is used to optimize the depth map to obtain the optimized depth map;

Step S200: Use the optimized depth map to establish a local pixel model of the current frame of the stereo camera image, and compare the conflict information between the local pixel model of the current frame and the global pixel model corresponding to all image frames before the current frame. Eliminate, and use the local segment model of the current frame after eliminating conflict information to fuse with the global scene element model until the global scene element model corresponding to all image frames of the autonomous driving scene is constructed;

Step S300: Perform adaptive rendering on the global scene element model corresponding to all image frames of the autonomous driving scene to obtain an adaptively rendered global scene element model, and obtain information related to the autonomous driving scene from the adaptively rendered global scene element model. The corresponding new perspective scene image sequence; wherein the new perspective scene image sequence includes multiple new perspective scene rendering images;

Step S400: Obtain the rendering image pyramid of the new perspective scene rendering image, the reference image pyramid of the reference image corresponding to the new perspective scene rendering image, and the mask image pyramid of the mask image corresponding to the new perspective scene rendering image, and then render the image pyramid , input the trained adversarial generative network model with reference image pyramid and mask image pyramid, and use the adversarial generative network model to complete the missing information areas in the new perspective scene rendering image to generate a complete new perspective scene image sequence.

Among them, the complete new perspective scene image sequence in the above step S400 includes multiple complete new perspective scene images; the reference image is a stereo camera image from an adjacent perspective of the new perspective scene rendering image, and the mask image is based on the new perspective scene rendering image. obtained from areas where information is missing.

The above-mentioned stereo camera image is obtained by shooting the autonomous driving scene with a stereo camera. Contextual information is used to provide contextual matching information for areas where information is missing. The depth map is used to represent the depth information of the stereo camera image, and the semantic map is used to represent the semantic information of the stereo camera image; the semantic map is used to optimize the depth map to obtain an optimized depth map.

The above complete new perspective scene image sequence includes multiple complete new perspective scene images corresponding to the autonomous driving scene. The reference image is used to provide scene context information for the new perspective scene rendering image.

In the above-mentioned step S100, the stereo camera image is obtained by photographing the aforementioned automatic driving scene by a stereo camera. Among them, those skilled in the art can select autonomous driving scenarios according to actual needs, and the specific content included in the autonomous driving scenarios is not limited here.

In some embodiments, in the above step S100, the existing semantic segmentation model can be used to process the aforementioned stereo camera image to generate a semantic map of the autonomous driving scene. Among them, the semantic segmentation model can be a pre-trained model of PSM-Net, etc. Among them, PSM-Net (pyramid stereo matching network) is a network composed of spatial pyramid pooling and three-dimensional convolution layers. It incorporates global background information into stereo matching to achieve occlusion areas, no texture or repetition. A reliable estimate of the area. Spatial pyramid pooling can obtain global context information through multi-scale accumulation, and a network composed of three-dimensional convolutional layers can obtain disparity maps.

It should be noted that the specific process of using the existing semantic segmentation model to process the aforementioned stereo camera image to generate the semantic map of the autonomous driving scene in the above step S100 belongs to the existing technology in this field, so the specific process will not be discussed here. Elaborate.

In some embodiments, in the above step S100, the existing depth prediction model can be used to process the aforementioned stereo camera image to generate a depth map of the autonomous driving scene. Among them, the depth prediction model can be PointRend's pre-trained model, etc. PointRend (point-based rendering) neural network is based on an iterative segmentation algorithm that performs point-based segmentation prediction at adaptively selected locations; by building on the latest existing models, PointRend can be flexibly applied to instances and semantics Split tasks.

It should be noted that the specific process of using the depth prediction model to process the aforementioned stereo camera image to generate a depth map of the autonomous driving scene in the above step S100 belongs to the existing technology in this field, so the specific process will not be described again here. .

It should be noted that the purpose of using the existing semantic segmentation model to generate the above-mentioned semantic map in the above-mentioned step S100 is as follows: first, using the above-mentioned semantic map to optimize the less accurate boundary depth information of the object in the above-mentioned depth map; That is, optimize the boundary depth information of all objects in the above-mentioned depth map; the second is to use the above-mentioned semantic map to eliminate dynamic objects in the above-mentioned depth map.

Due to the uncertainty of the prediction algorithm (such as the depth prediction algorithm used by the aforementioned depth prediction model), outliers or noise will inevitably be introduced into the aforementioned depth map. For example, under normal circumstances, different types of objects have different depths (such as trees, houses, cars, etc.), but the depth information output by the depth prediction model has a relatively large error in estimating the depth of the boundary. Therefore, in the above step S100, "using the semantic map to Optimize the depth map to obtain an optimized depth map. "Optimize the boundary depth information of all objects in the aforementioned depth map with the help of more accurate semantic information (i.e., the aforementioned semantic map), that is, enhance it by optimizing the boundaries of the target objects in the aforementioned depth map. The aforementioned depth map is then used to obtain the optimized depth map. Specifically, first determine the boundary of the target object in the depth map that needs to be optimized, that is, the area in the depth map where the depth gradient change exceeds a preset threshold; generally, the closer the area is to the target boundary, the greater the change in depth gradient. Large; after that, the corresponding depth is optimized to target points that are geometrically close to the semantic boundaries. According to the depth locality principle of semantically consistent regions, each pixel depth can be optimized by averaging the effective parts of N pixel blocks around it, while ensuring that the depth value of the above depth map is fused by semantically identical pixels and depth boundaries. Semantic boundary direction optimization. In some embodiments, points in the depth map whose adjacent depths are very different can also be filtered out. Dynamic objects such as pedestrians and cars can be removed from the above depth map by comparing sequential frames and semantic labels. Finally, the boundary abnormal points in the above depth map are optimized and the dynamic targets are removed to obtain the optimized depth map.

The surface of a three-dimensional surface element model (such as the aforementioned global surface element model and local surface element model) is quantized as a set of oriented small disks, namely surface elements. The size of the bin is determined by the resolution of the image. Each surfel is effectively defined and visualized with multiple attributes, such as position, normal, radius, color, semantics, and depth information. The aforementioned surface element model uses surface elements to perform continuous or discontinuous geometric description and feature description of the surface of the three-dimensional space object without studying the internal characteristics of the three-dimensional space object.

Although the depth map is optimized in step S100 above, before the reconstructed new local model (i.e., the above-mentioned initial local pixel model) is merged into the existing global model (i.e., the above-mentioned initial global pixel model), it is still necessary to Further denoising is performed on the above initial local surface element model. Therefore, in the above step S200, "the optimized depth map is used to establish an initial local surface element model, and the conflict information in the initial local surface element model that conflicts with the initial global surface element model is eliminated, and the conflict information after eliminating the conflict information is used. The specific process of "the local surface element model fuses the initial global surface element model to obtain the fused global surface element model" is discussed below. Since the performance of depth prediction degrades when the target is further away, depth measurements closer to the optical center of the camera (such as the aforementioned stereo camera) are more accurate. Based on this assumption, the depth in the existing initial local block model is smaller than the mapping depth of the initial global block model, that is, the local block is closer to the camera optical center than the global block. This is due to the camera moving toward the target and the field of view. Less than 180 degrees. If within a certain threshold range, the depth of the local surface element is farther from the optical center of the camera than the depth of the global surface element in the global surface element model is from the optical center of the camera, set it to a lower confidence level, otherwise set it to for a higher confidence level.

In some embodiments, "surface elements with relatively large depth conflicts" (i.e., the above-mentioned conflict information) can be found through mapping. After eliminating the conflict information in the above-mentioned initial local surface element model, it is necessary to use the local area elements after eliminating the conflict information. The surface element model fuses the initial global surface element model. In some embodiments, there is no confidence in the predicted depth of local bin elements, or the local bin points are much farther from the camera optical center than the global bin points (such as exceeding a preset threshold). These local bin points will be used as conflicts. The points are deleted, and the global element model is further fused to obtain the fused global element model.

It should be noted that the above-mentioned depth confidence can be directly initialized using existing technology. For example, the above initialization can be adopted from the existing literature "M.F.A. Eldesokey and F.S. Khan, "Confidence propagation through cnns for guided sparse depthregression," IEEE transactions on pattern analysis and machine intelligence, vol.42, no.10, pp.2423–2436, 2019" method of recording. Since the above initialization can directly adopt the existing technology in this field, it will not be described again here.

It should be noted that the reconstruction of the above-mentioned local bin model is based on the depth information (i.e., the aforementioned depth map). Therefore, after the conflict depth (i.e., the above-mentioned conflict information) in the initial local bin model is eliminated, the remaining local The surface element model (this application considers the remaining local surface element model to be relatively accurate) is then integrated into the above-mentioned global surface element model.

In some embodiments, when "using the local block model of the current frame after eliminating conflict information and the global block model to fuse" in the above step S200, in order to avoid outliers in the model fusion process, only those local block models are Only local polygons that can match global polygons with similar color and semantic attributes and high confidence will be integrated into the global polygon model. When a local bin element is closer to the optical center of the camera than a global bin element, only the local bin element is retained. If both the local surface element and the global surface element are within the preset distance threshold, the aforementioned confidence weighted average method is used to fuse the global surface element and the local surface element. If there is no pairing of local bins and global bins, the new local bins will be added to the aforementioned global bin model as unstable bin elements, and their confidence level will also be set to a relatively low value.

It should be noted that an autonomous driving scene includes image frames of multiple stereo camera images, for example, multiple consecutive image frames. In the above step S200, the "global pixel model corresponding to all image frames before the current frame" is fused frame by frame. That is, after the local pixel model of the current frame is constructed, the local pixel model of the current frame is constructed. The conflict information that conflicts with the meta model and the global scene element model corresponding to all image frames before the current frame is eliminated, and then the local face element model corresponding to the current frame is integrated into the global scene element corresponding to all image frames before the current frame. model; and by analogy, as the number of fused frames increases, the global element model will become larger and larger until the global element model corresponding to all image frames of the autonomous driving scene is constructed. It should be noted that when constructing the local pixel model of the first frame (i.e., the first current frame), there is no global pixel model; the global pixel model is derived from the second frame (i.e., the second current frame). The initial construction is to use the local surface element model corresponding to the first current frame as the above-mentioned global surface element model corresponding to all image frames before the current frame, and so on. Until the last frame of the autonomous driving scene (i.e., the last current frame) is integrated into the aforementioned global scene element model corresponding to all image frames before the current frame, at this point, the global scene element model corresponding to all image frames of the autonomous driving scene The build is complete. Among them, all image frames before the current frame correspond to the above-mentioned stereo camera images.

In some embodiments, in the above step S300, the new perspective scene image sequence corresponding to the autonomous driving scene can be directly obtained from the global scene element model corresponding to all image frames of the autonomous driving scene.

In some embodiments, in the above step S300, the global scene element model corresponding to all image frames of the autonomous driving scene can be adaptively rendered to obtain an adaptively rendered global scene element model. From the adaptively rendered global scene element model, The meta-model obtains a new perspective scene image sequence corresponding to the autonomous driving scene.

It should be noted that the specific process of obtaining the new perspective scene image sequence corresponding to the autonomous driving scene from the above-mentioned global scene element model belongs to the existing technology and common knowledge in this field, so it will not be described again here.

It should be noted that the perspective captured by the aforementioned camera is generally called the "original perspective". In the process of reconstructing the 3D panel model (such as the global panel model), it is also established based on the "original perspective". After the 3D panel model is established, , you can collect "new perspective" (i.e., "non-original perspective") pictures from the 3D surface element model, and collect them continuously like videos to obtain a sequence of new perspective scene images.

After the foregoing global scene element model is constructed, a new perspective image sequence can be obtained from the incomplete aforesaid global scene element model. The camera's new viewpoint is close to the original viewpoint, where, in some embodiments, the optical axis is limited to ±15 degrees and the baseline is relatively wide (approximately ±1 m). In some embodiments, in order to better utilize the high-fidelity visualization of the aforementioned global scene element model, this application provides a processing method for adaptive rendering of the global scene element model corresponding to all image frames of the autonomous driving scene. This processing method includes two main solutions, which are used to adjust the normals and radii of the surface elements in the aforementioned global surface element model. First, the surface elements of the far viewpoint in the aforementioned global surface element model will be adjusted according to the normal vector along the observation direction. This operation ensures that distant surface element points can effectively participate in rendering. Second, a specific radius in the aforementioned global element model tends to produce blurred textures at shorter distances from a new perspective. To overcome this problem, in some embodiments, the surf radius may be adjusted using the angle between the normal of the surface element and the observation direction of the new perspective. When a bin point covers multiple pixels and the normal is within a certain range of the observation direction (such as π/2), in order to avoid redundant overlap with other bin points, its radius can be greatly reduced from r _i (u) to . Among them, u is the position of the pixel covered by the bin, α is the angle between the normal of the bin and the observation direction of the new perspective, and i here represents the number of the bin. Therefore, in some embodiments, the normals and/or radii of the surface elements in the global element model (such as the surface elements of the far viewpoint) can be adjusted according to the normal vector along the above-mentioned observation direction.

The above-mentioned step S400 is mainly used to complete the image completion of the traffic scene (that is, the above-mentioned new perspective scene rendering image). The above global element model is built based on real-world sensor data (such as the above-mentioned stereo camera images). Due to the incomplete reconstruction of the above-mentioned global element model and the removal of the above-mentioned dynamic objects, various masks or holes appear in the scene background rendering image (that is, the above-mentioned new perspective scene rendering image). This application mainly considers the consistency of space and content, uses certain content to fill these masks/holes (that is, the areas with missing information in the aforementioned new perspective scene rendering image), and uses the adversarial generative network model below to synthesize high-quality, Realistic traffic scene sequence (that is, the complete new perspective scene image sequence mentioned above). The inputs of the adversarial generative network model include new perspective scene rendering images of different scales (that is, different resolutions), mask images of different scales, and reference images of different scales. The overall processing process of the adversarial generative network model includes two stages:

In the first stage, the new perspective scene rendering image of the corresponding scale is first divided into multiple rendering sub-blocks in the form of sub-blocks (patch). The rendering sub-blocks are searched for approximate context on the reference image through the mask image representation association mechanism below. To obtain context-related information; in order to increase the matching accuracy of the above-mentioned approximate context-related information, the mask sub-block of the corresponding mask image is applied to the reference sub-block corresponding to the reference image to obtain the reference mask sub-block, and then rendering is performed The sub-block is matched to the reference mask sub-block;

In the second stage, rendering images of multiple scales are used as input, and the adversarial generative network model is used to complete the missing information areas (such as holes, etc.) in the new perspective scene rendering image to generate a consistent, high-quality complete image with a consistent spatial structure. New perspective scene image sequence; where the approximate context-related information predicted in the first stage (i.e., the associated images and similarity maps below) is transferred to multiple CSD sub-modules to maintain the complete generated new perspective scene image sequence. consistency between. Through the above two stages of image completion processing, the generated traffic scene images (that is, the complete new perspective scene image sequence) are more realistic and more coherent in content.

It should be noted that, please refer to Figure 4. The mask image Im is obtained based on the area with missing information in _{the new} perspective scene rendering image Ir . This is because the new perspective scene rendering image I _r obtained from the global scene element model after adaptive rendering itself will have various irregular holes/masks. Similarity matching of a new perspective scene rendering image (i.e. reference image) is likely to result in low accuracy of similarity matching (for example, there are holes in an area of the new perspective scene rendering image that need to be completed, and the images from adjacent perspectives are This hole does not exist in the same area of the reference image. If similarity matching is performed directly, it is understandable that it is difficult to match similar context information in this area). Therefore, various aspects of the new perspective scene rendering image I _r can be first The irregular holes/mask are extracted separately and made into a mask _{image Im corresponding} to the new perspective scene rendering image Ir (for example, there are irregular holes in the rearview mirror area of the car in _the new perspective scene rendering image Ir , this To apply, the area with irregular holes in the rearview mirror area of the car is extracted separately to generate a mask image Im ₎ , and then _the mask image Im is applied to the above reference image to obtain the corresponding reference mask sub-block. In this way, the same irregular holes/masks and other areas with missing information can automatically appear in _the reference image corresponding to various irregular holes/masks in the new perspective scene rendering image Ir , which can significantly improve The matching accuracy of the above approximate context-related information (such as the accuracy of matching the rendering sub-block and the reference sub-block below); conversely, if the mask image Im is not applied to the reference image and the corresponding reference mask sub-block _is obtained and the reference is directly used Similarity matching between the mask sub-block and the rendering sub-block will easily lead to low accuracy of the subsequently generated associated image I _c' , which will make it impossible to complete the missing information areas in the new perspective scene rendering image with high quality.

Please refer to Figure 2. In the above-mentioned step S400, the rendered image pyramid, the reference image pyramid and the mask image pyramid are input to the trained adversarial generative network model to use the adversarial generative network model to perform analysis on the areas with missing information in the new perspective scene rendering image. Completion to generate a complete new perspective scene image sequence, including:

Step S410: The image results of each layer in the rendered image pyramid, the reference image pyramid and the masked image pyramid are respectively input into the first-level CSD residual module to the last-level CSD residual module in the generator in order from the top to the bottom. ; Use the output of the CSD residual module of the previous level as an input of the adjacent CSD residual module of the next level, and use the output of the CSD residual module of the last level as a complete new perspective scene image; where, generate The network structures of the first-level CSD residual module to the last-level CSD residual module in the device are all the same; among them, the number of layers of the rendering image pyramid, reference image pyramid and mask image pyramid is the same as the number of CSD residual modules. same.

In some embodiments, in the above-mentioned step S400, the new perspective scene rendering image, the reference image corresponding to the new perspective scene rendering image, and the mask image corresponding to the new perspective scene rendering image can be separately down-sampled or other existing techniques. Obtain a rendered image pyramid for the new perspective scene rendered image, a reference image pyramid for the reference image, and a mask image pyramid for the mask image.

The image pyramid is a collection of images with different resolutions, mainly using two common operations: upsampling and downsampling. Downsampling refers to sampling from a high-resolution image to a low-resolution image and performing Gaussian filter smoothing. Upsampling refers to interpolating from a low-resolution image to a high-resolution image and performing Gaussian filter smoothing. The resolution of the image obtained by upsampling the original image becomes twice that of the original image. The resolution of the image obtained by downsampling the original image becomes half that of the original image.

In some embodiments, the brief process of downsampling to build an image pyramid can be as follows: first perform a Gaussian filter smoothing process on a given image, that is, perform a convolution operation on the above image; perform downsampling on the above image, where , you can take odd columns in the row direction of the above image, and take even columns in the column direction of the above image; or you can take even columns in the row direction of the above image, and take odd columns in the column direction of the above image; for the sampled image , repeat the first two steps to obtain the image pyramid. It should be noted that in some embodiments, the new perspective scene rendering image, the reference image and the mask image are respectively down-sampled to obtain the rendering image pyramid of the new perspective scene rendering image, the reference image pyramid and the mask image of the reference image respectively. The specific process of masking the image pyramid belongs to the existing technology in this field, so it will not be described again here.

In some embodiments, in the above-mentioned step S400, the scale (or spatial resolution) of the image results of each layer in the above-mentioned rendered image pyramid/reference image pyramid and mask image pyramid may be 1/32, 1 in order from the top to the bottom. /16, 1/8, 1/4, 1/2, where 1/2 means that the scale (or spatial resolution) of the current layer is 1/2 of the original scale (or spatial resolution), and so on. Those skilled in the art can adjust the scale (or spatial resolution) of the image results of each layer in the above-mentioned rendered image pyramid/reference image pyramid and mask image pyramid according to actual needs.

In some embodiments, the adversarial generative network model includes a generator. Please refer to Figure 7. The generator includes multiple levels of CSD residual modules connected in series (the CSD residual modules from left to right in Figure 7 are the first-level CSD residual module and the second-level CSD residual module. to the last level CSD residual module). Among them, CSD (Contextual and Spatial Denormalization) represents contextual and spatial denormalization.

Please refer to Figure 7. In the above step S410, the image results of each layer in the rendering image pyramid, the reference image pyramid and the mask image pyramid are respectively input to the CSD residual module of the first level in the generator to the end in order from the top to the bottom. The first-level CSD residual module; the output of the previous-level CSD residual module is used as an input of the adjacent next-level CSD residual module, and the output of the last-level CSD residual module is used as a complete new Perspective scene images, including:

Input the top-level image results in the rendered image pyramid, reference image pyramid and mask image pyramid, as well as the new perspective scene rendering image corresponding to the scale of the top-level image result, into the first-level CSD residual module;

The image results of the remaining layers in the rendered image pyramid, reference image pyramid and masked image pyramid except the top layer are respectively input into the second-level CSD residual module in the generator in order from the next layer of the top layer to the bottom layer to the end. The first-level CSD residual module; the output of the last-level CSD residual module is used as a complete new perspective scene image.

In some embodiments, the image results of the top layer in the rendered image pyramid, the reference image pyramid and the mask image pyramid, as well as the batch of new perspective scene rendering images corresponding to the scale of the top layer of image results, can be input into the first-level CSD residual. module. Those skilled in the art can determine the quantity of each batch (batchsize=N) mentioned above by themselves. For example, the quantity of each batch is five.

The above-mentioned CSD residual module includes a first-level CSD sub-module, a second-level CSD sub-module and a third-level CSD sub-module. Among them, CSD (Contextual and Spatial Denormalization) sub-module, that is, contextual and spatial denormalization sub-module.

Please refer to Figure 3. The data processing flow inside the above CSD residual module is:

The image results of the corresponding layers in the rendered image pyramid, reference image pyramid and mask image pyramid _{(please refer to Figure 4, such as the corresponding scale mask image Im} , reference image Ic _and new perspective scene rendering image Ir ₎ , and The feature map input to the CSD residual module is input to the first-level CSD sub-module and the third-level CSD sub-module respectively. The output of the first-level CSD sub-module is used as the input of the first ReLU activation function. The output of the activation function is convolved to obtain the first convolution result. The first convolution result is input to the second-level CSD sub-module, and the output of the second-level CSD sub-module is used as the input of the second ReLU activation function. Perform a convolution operation on the output of the second ReLU activation function to obtain the second convolution result, use the output of the third-level CSD sub-module as the input of the third ReLU activation function, and perform convolution on the output of the third ReLU activation function. The operation is performed to obtain the third convolution result, and the unit addition operation is performed on the second convolution result and the third convolution result to obtain the output of the CSD residual module;

Among them, the feature map input to the CSD residual module of the first level is the new perspective scene rendering image corresponding to the scale of the top-level image result (that is, the new perspective scene rendering image that needs to be completed), and the input of the adjacent subsequent level is The feature map of the CSD residual module (please refer to x ⁱ in Figure 3) is the output of the CSD residual module of the previous stage.

For example, the feature map input to the second-level CSD residual module is the output of the first-level CSD residual module, and the feature map input to the third-level CSD residual module is the output of the second-level CSD residual module. Input the feature maps of other levels of CSD residual modules and so on.

In some embodiments, the number of layers of the rendered image pyramid, the number of layers of the reference image pyramid, and the number of layers of the mask image pyramid are the same as the number of CSD residual modules in the generator. The above-mentioned first ReLU activation function, second ReLU activation function and third ReLU activation function are currently commonly used ReLU activation functions in this field.

The data processing procedures of the above-mentioned first-level CSD sub-module, second-level CSD sub-module and third-level CSD sub-module are all the same. Please refer to Figure 4. BatchNorm in Figure 4 represents the normalization layer or normalization operation, conv represents the following convolution operation (such as the first convolution operation), MPC represents the mask image representation association mechanism below, element -wise means element-wise multiplication operation.

Please refer to Figure 4. The data processing flow of the CSD sub-module is:

The image results of the corresponding layer in the input rendered image pyramid (such as the new perspective scene rendering image I _r corresponding to the scale), the image results of the corresponding layer in the reference image pyramid (such as the reference image I _c corresponding to the scale) and the mask image pyramid The image results of the corresponding layer in the image pyramid (such as the mask image I _m of the corresponding scale) are subjected to similarity matching to obtain context-related information with the image results of the corresponding layer in the rendered image pyramid;

Among them, the context associated information includes the similarity map and the associated image I _c' of the image result of the corresponding layer in the rendered image pyramid (ie, the corresponding scale I _r ); the associated image is based on the image result of the corresponding layer in the reference image pyramid (ie, the corresponding The I _c of the scale is generated by multiple regions that are most similar to the image results of the corresponding layer in the rendered image pyramid (i.e., I _r of the corresponding scale). The similarity map is used to characterize the image results of the corresponding layer in the rendered image pyramid (i.e. The correlation between I _r at the corresponding scale) and the image result of the corresponding layer in the reference image pyramid (i.e. I _c at the corresponding scale);

Among them, context-related information is used to provide context matching information for areas with missing information;

Perform a second convolution operation on the associated image I _c' and the similarity map respectively to obtain the spatial features f ( I _c' ) of the associated image I _c' and the spatial features of the similarity map respectively;

Perform a second convolution operation on the image result I _r of the corresponding layer in the rendered image pyramid to obtain the spatial feature f ( I _r ) of the new perspective scene rendering image of the corresponding layer;

Perform element-wise multiplication operations on the spatial features α _{c, h, w} of the similarity map and the spatial features f ( I _r ) of the new perspective scene rendering image to obtain the element-wise multiplication result;

Add the element-wise multiplication result to the spatial feature f ( I _c' ) of the associated image I _c' to obtain the second affine parameter β _c,h,w ; where the expression of the second affine parameter is:

; is the element-wise multiplication result, α _{c, h, w} are the spatial characteristics of the similarity map, is the spatial feature of the associated image I _c' ; wherein, the c, h and w are the number of characteristic channels, the length and the width of the image result of the corresponding layer respectively, and the I _r represents the new perspective scene rendering image, The I _c' is the associated image of the image result of the corresponding layer;

Normalize multiple feature maps input to the CSD residual module to obtain normalized parameters;

The normalized result obtained by adding the normalized parameter and the second affine parameter β _{c, h, w} (ie, BN ^' ( xi ⁾ below) is used as the output of the CSD sub-module.

It should be noted that performing the second convolution operation on the associated image I _c' and the similarity map are respectively implemented by different simple two-layer convolution networks. Performing the second convolution operation on the image result of the corresponding layer in the rendered image pyramid (ie, I _r ) is also implemented by a simple two-layer convolutional network. However, it is important to note that the simple two-layer convolutional network corresponding to the image result of the corresponding layer in the rendered image pyramid and the simple two-layer convolutional network described above corresponding to the associated image operate the same and share parameters.

In some embodiments, please refer to Figure 5 to perform similarity matching on the image results of the corresponding layer in the input rendered image pyramid, the image result of the corresponding layer in the reference image pyramid, and the image result of the corresponding layer in the mask image pyramid to obtain Contextual information related to the image results of the corresponding layer in the rendered image pyramid, including:

Use a dividing frame to move in the image result of the corresponding layer in the rendered image pyramid (i.e., I _r of the corresponding scale) in the first step to divide the image result of the corresponding layer in the rendered image pyramid into multiple renderings of size k×k sub-block,

The dividing frame is used to move in the image result of the corresponding layer in the mask image pyramid (i.e., I _m of the corresponding scale) in the first step to divide the image result of the corresponding layer in the mask image pyramid into multiple images of size k×k. mask sub-block,

Determine the search domain corresponding to the rendering sub-block in the image result of the corresponding layer in the reference image pyramid (i.e., I _c of the corresponding scale). The size of the search domain is k'×k', and the dividing box is used to search in the second step The search domain is divided into s×s reference sub-blocks of size k×k by moving in the domain; k, k' and s are all preset constants;

Apply the mask sub-block corresponding to the rendering sub-block to the reference sub-block corresponding to the rendering sub-block to obtain a reference mask sub-block corresponding to the rendering sub-block;

Calculate each similarity between the rendering sub-block and each reference mask sub-block in the search domain corresponding to the rendering sub-block, and use the maximum value of each similarity as the element corresponding to the rendering sub-block in the similarity map;

Among them, the size of the search domain corresponding to the rendering sub-block is larger than the size of the rendering sub-block;

The reference sub-block corresponding to each element in the similarity map is regarded as the best matching sub-block, and multiple best-matching sub-blocks corresponding to the similarity map are used to generate the associated image (ie, I _c' ).

In some embodiments, when k=3, the second stride m=1, s=3, and the size of the search domain k'×k'=[k+m×(s-1)]×[k+m ×(s-1)], so the above k'=5. Those skilled in the art can adjust the above parameters k, m, s and k' according to actual needs.

In some embodiments, in the above process of obtaining contextual information related to the image result of the corresponding layer in the rendered image pyramid, the dividing box is a virtual box. It can be understood that the above-mentioned division box can be regarded as a convolution kernel, and the above-mentioned division box can be moved in the image result of the corresponding layer in the rendering image pyramid (ie, the corresponding scale I _r ) according to the first step (for example, from a row) from the far left to the far right, and then from the previous row to the next row, etc.), and the area of the image result of the corresponding layer in the rendered image pyramid surrounded by the above-mentioned dividing box is used as a rendering sub-block, and then the rendering The image result of the corresponding layer in the image pyramid is divided into multiple rendering sub-blocks of size k×k.

It should be noted that those skilled in the art can adjust the size of the first step n according to actual needs, and the size of the first step n is not limited here. For example, the first step n=1.

It should be noted that in the above-mentioned process of obtaining the context-related information of the image result of the corresponding layer in the rendered image pyramid, "the mask sub-block corresponding to the rendering sub-block is applied to the reference sub-block corresponding to the rendering sub-block to obtain the corresponding rendering sub-block. "Reference mask sub-block corresponding to the sub-block" refers to using the corresponding mask sub-block to block the corresponding reference sub-block to generate a reference sub-block with a mask (i.e. the above-mentioned reference mask sub-block), and then calculate the rendering sub-block The similarity between multiple reference mask sub-blocks within the search domain of this rendering sub-block.

As mentioned above, the new perspective scene rendering image obtained from the above global element model is incomplete, but there are one or more original real images corresponding to the new perspective scene rendering image at the same acquisition perspective. In some embodiments, the above Im _∈ L ^{H × W} is defined as a binary mask image (ie, the above mask image), where H and W are respectively the height and width of the mask image corresponding to the new perspective scene rendering image. Please refer to Figure 5. The reference image I _c , that is, the condition sample, is the original image (ie, the aforementioned stereo camera image) collected by the camera (such as the aforementioned stereo camera). The reference image I _c comes from the neighboring viewpoint of the new perspective scene rendering image I _r . The reference image I _c provides scene context information for the completion of the new perspective scene rendering image I _r . The correlation calculation method between the new perspective scene rendering image I _r and the corresponding reference image I _c is similar to convolution. The input new perspective scene rendering image I _r is divided into multiple rendering sub-blocks of size k×k. Its mask image is divided accordingly into the same set of mask sub-blocks. Each rendering sub-block of the new perspective scene rendering image I _r operates as a convolution kernel in the corresponding search domain in the conditional sample. The size of the search domain is larger than the size of the rendering subchunk. Before matching, all reference sub-blocks must be assigned corresponding binary masks (ie, mask sub-blocks) to improve the matching accuracy of the above correlation calculation.

It should be noted that the reason why the size of the above search domain is larger than the size of the rendering sub-block is because there is a deviation in the perspective between the "new perspective scene rendering image I _r " and the "original perspective", so it needs to be searched when doing context matching. A larger area can match the corresponding contextual information.

The expression of the above similarity graph is:

;

Where, simi ( i ) represents the i -th rendering sub-block in the image result of the corresponding layer in the rendering image pyramid and s×s reference mask sub-blocks in the search domain corresponding to the i-th rendering sub-block in the image result of the corresponding layer in the reference image pyramid (i.e. I _c of the corresponding scale)/> The maximum value of each similarity between them,/> To obtain the cosine similarity operation, ⊙ represents element-wise multiplication, where , where H and W are the height and width of the new perspective scene rendering image respectively, n is the first step above; I _m represents the mask image, I _c represents the reference image, I _r represents the new perspective scene rendering image, and j represents Search reference sub-blocks within the domain/> quantity,/> is the mask sub-block.

The expression of the above normalized parameters is: , where x ⁱ is a batch of feature maps (such as N feature maps) of the CSD submodule in the i-th level CSD residual module input, x ⁱ ∈R ^N×C×H×W , and C is the corresponding feature channel quantity), for example, x ⁱ can refer to the feature map of the first-level CSD sub-module and the third-level CSD sub-module in the input i-th level CSD residual module; μ _c ( x ⁱ ) and σ _c ( x ⁱ ) are respectively the mean and standard deviation of the N feature maps of the CSD submodule in the CSD residual module input to the i-th level, γ _c is the first affine parameter, γ _c ∈ R ^C ; the modulated affine parameters include the first affine parameter The radial parameter γ _c and the second affine parameter β _c,h,w . Among them, the expressions of the mean and standard deviation are:

,

; Among them, n∈N, c∈C, h∈H, w∈W, and ε here are preset coefficients, and the settings of the preset coefficients are common knowledge in this field.

In some embodiments, those skilled in the art can flexibly set the size (i.e. size) of the above-mentioned search domain according to actual needs. For example, the size of the search domain can be 5x5, in which case the search domain is based on the reference image I _c and The reference mask sub-block corresponding to this rendering sub-block is an area with a center and a side length of 5. Normally, the values of H and W mentioned above are the same.

It should be noted that, in order to improve the accuracy of similarity calculation through the expression of the above similarity map, it is necessary to apply the mask sub-block corresponding to the above mask image to each reference sub-block in the search domain in the reference image to obtain each reference The reference mask sub-block corresponding to the sub-block (i.e., the above Indicates that the i- th mask sub-block is assigned to each reference sub-block in the search domain corresponding to the i- th rendering sub-block in the reference image I _c , and then the corresponding reference mask sub-blocks in the search domain are calculated). The similarity between the mask sub-block and the corresponding rendered sub-block.

In some embodiments, the index of the best matching sub-block may be recorded. The best matching sub-block refers to the reference sub-block corresponding to each element in the similarity map, that is, the maximum value of each similarity between the rendering sub-block and multiple reference mask sub-blocks in the search domain corresponding to the rendering sub-block. The corresponding reference sub-block. The above index is used to mark the corresponding position of the best matching sub-block in the search domain corresponding to the above-mentioned rendering sub-block. For a large area with missing information in the new perspective scene rendering image I _r , since the reference sub-block corresponding to this area is completely filled by the mask sub-block, the similarity between the above-mentioned rendering sub-block and multiple reference mask sub-blocks is in the middle One or more of may be zero, resulting in subsequent incorrect associations. In order to alleviate the above technical problem, in some embodiments, when determining the index of the best matching sub-block corresponding to each element in the similarity graph one by one (please refer to element S1 to element S9 in Figure 5), it is assumed that a The search domain corresponding to the rendering sub-block contains 9 reference mask sub-blocks. Please refer to Figure 6. a1 to a9 can be regarded as the similarity between the rendering sub-block and the 9 reference mask sub-blocks in the search domain corresponding to the rendering sub-block. value, this application uses the average of the similarity values corresponding to the non-zero index to replace the similarity value of the zero-value index. For example, if the calculated a1 to a9 are all 0 (wherein, the corresponding indexes of a1 to a9 are 0 to 8), and the zero value indexes are 0 to 8 respectively, then the rendering sub-block and the rendering sub-block The maximum value of similarity between reference mask sub-blocks in the corresponding search domain is 0. In this case, by default the best matching sub-block of the rendering sub-block is the 0th reference mask sub-block in the search domain. The reference sub-block is a1, but the reference sub-block corresponding to a1 is not actually the best matching sub-block. Therefore, this application uses the similarity values corresponding to certain non-zero indexes around the zero-value index (for example, the non-zero index can be the four adjacent indexes above, below, left and right of the zero-value index). Calculate the average value and use this average value to replace the similarity value corresponding to the zero-value index. Then, based on the maximum value of each similarity between the reference mask sub-blocks in the search domain corresponding to the rendering sub-block, Index to find the best matching sub-block. For example, the calculated index of the best matching sub-block is 5, and the index value 5 refers to a6, that is, the best matching sub-block of the rendering sub-block is the one corresponding to a6 in the search domain corresponding to the rendering sub-block. Reference sub-block corresponding to the reference mask sub-block. Those skilled in the art can adaptively adjust the specific determination process of the best matching sub-block according to actual needs, such as flexibly setting the specific number of non-zero indexes around the zero-valued index.

In some embodiments, in the above-mentioned process of obtaining contextual information related to the image result of the corresponding layer in the rendered image pyramid, "the reference sub-block corresponding to each element in the similarity map is used as the best matching sub-block, and the similarity is used to The process of "generating associated images from multiple best matching sub-blocks corresponding to the graph" can be: the first step is to obtain the best matching sub-blocks corresponding to each element in the above-mentioned similarity graph through the above-mentioned index, where these best matches The sub-blocks are all independent of each other; in the second step, these best-matching sub-blocks need to be spliced into a picture; the specific splicing method is the same as the deconvolution method, so the best-matching sub-block can be called here is the "deconvolution filter"; in the third step, during the process of piecing back a picture, partially overlapping areas will appear. The corresponding multiple pixel values in these partially overlapping areas are averaged, and the calculation is The average value is used as the value of the corresponding pixel point in the above-mentioned partially overlapping area, and then the associated image I _{c '} corresponding to the above-mentioned reference image I c is finally obtained.

It should be noted that the specific process of splicing the above-mentioned best matching sub-blocks into a picture using deconvolution belongs to the existing technology in this field, so the specific process will not be described again here.

In some embodiments, please refer to Figure 3. The above-mentioned CSD residual module includes a first-level CSD sub-module, a second-level CSD sub-module and a third-level CSD sub-module.

Given the scene context information (i.e., the above-mentioned conditional sample), this application normalizes the layers in the above-mentioned first-level CSD sub-module, second-level CSD sub-module and third-level CSD sub-module. Contextual information (that is, the above-mentioned corresponding associated images and corresponding similar images) is added to fill in the areas with missing information in the above-mentioned new perspective scene rendering images.

It should be noted that the above-mentioned given scene context information and context-related information can provide context matching information, so that the adversarial generative network model can calculate the specific areas with missing information in the new perspective scene rendering image.

In some embodiments, please refer to Figure 4. The inspiration for adding context-related information to the above-mentioned normalization layer comes from conditional batch normalization (BatchNormalization, BN), whose activation layer is normalized to zero mean and unit deviation. Denormalization is then performed by adjusting affine parameters from external data. Modulated affine parameters corresponding to the scale can be used to control the global output of the adversarial generative network model while preserving the spatial structure of the image content, as also demonstrated by adaptive instance normalization (AdaIN). It should be noted that the global output can include the content and style of the final output image of the adversarial generative network model (such as a complete new perspective scene image sequence). Each normalization layer in each CSD sub-module (ie, context and spatial denormalization sub-module) proposed in this application utilizes spatially varying affine transformations to make it suitable for image synthesis. Similar to batch normalization (BN), each CSD sub-module of this application normalizes the mean and standard deviation of each individual feature channel (n∈N, c∈C, h∈H, w∈W) . Among them, the expression of the above normalized result is:

. The above normalized results are used as the final output of each CSD sub-module.

It should be noted that the modulation affine parameters of all normalization layers in each CSD residual module (i.e., the above-mentioned modulation affine parameters corresponding to different scales) are learned from each CSD sub-module. This application inputs the new perspective scene rendering image I _r and the reference image I _c into the normalization layer of each CSD sub-module instead of the first layer network of the entire adversarial generation network model. This is because each CSD sub-module is smaller than the ordinary normalization layer. The unified layer can better preserve the input information. In fact, the modulation affine parameters learned by each CSD sub-module have already encoded enough input image information, so this application abandons the encoding part of the generator usually used in encoder-decoder generative adversarial networks.

The generator of this adversarial generative network model consists of several CSD residual modules (i.e., CSD-ResBlk) with upsampling layers. The output of each CSD residual module is the feature of the corresponding dimension.

It should be noted that the above-mentioned normalization layer specifically corresponds to the symbol "BatchNorm" in Figure 4.

This application first downsamples the input new perspective scene rendering image, reference image and corresponding mask image to establish their respective image pyramids, and then performs contextual information correlation on each CSD residual module to match the space at different scales. Resolution, the matching range can be flexibly adjusted by setting different sub-blocks (such as rendering sub-blocks and reference mask sub-blocks) and different search domains.

It should be noted that the above "by correlating contextual information for each CSD residual module, matching spatial resolution at different scales, by setting different sub-blocks (such as rendering sub-blocks and reference mask sub-blocks) and different searches Domain to flexibly adjust the matching range" operation is that the size of the above sub-blocks and search domains can be flexibly set. For example, when the scale of the image input is doubled, the sizes of the above sub-blocks and search domains remain unchanged. Therefore, the corresponding receptive field matched by the CSD residual module is doubled. This application hopes to expand the search domain as much as possible, so that it can handle scenes with large viewing angle deviations (such as turning scenes). However, such scenes require a large amount of calculation, so this application can flexibly adjust through inputs of different scales. The general steps of the above operation are: the first step is to scale the input image (i.e., the new perspective scene rendering image, the reference image, and the mask image) to different scales to obtain images of different scales (i.e., the image of the corresponding layer in the above-mentioned rendered image pyramid). result, the image result of the corresponding layer in the reference image pyramid and the image result of the corresponding layer in the mask image pyramid); in the second step, the above-mentioned pictures of different scales are input into the corresponding CSD sub-module to represent the association through the above-mentioned mask image mechanism (MPC) for context-related matching; in the third step, the obtained matching information is transmitted to the entire network of the adversarial generation network model through each CSD sub-module.

In some embodiments, the adversarial generative network model of the present application is similar to pix2pixHD, using a multi-scale discriminator/discriminator to train the generator. That is, the adversarial generative network model of this application can also include multiple discriminators and VGG convolutional network sub-models. Among them, pix2pixHD is an important upgrade of pix2pix, which can realize high-resolution image generation and semantic editing of pictures. For a Generative Adversarial Network (GAN), the key to learning is to understand the three parts of the generator, discriminator and loss function. The generator and discriminator of pix2pixHD are multi-scale, and the structure of the single-scale generator and discriminator is the same as that of existing pix2pix. pix2pix is an image translation model based on conditional generative adversarial network (CGAN), and conditional generative adversarial network is an extension of generative adversarial network. It achieves conditional translation by introducing conditional information in the generator and discriminator. Image generation. The generator of pix2pix adopts U-Net network structure to integrate low-level fine-grained features and high-level abstraction; the discriminator adopts patchGAN network structure to extract high-frequency information such as texture at the patch scale.

In addition, this application introduces the same feature matching loss in the discriminator to improve the adversarial loss and stabilize training; similar to the loss function of VGG, the perceptual loss L _VGG is jointly used to further improve performance. Therefore, the overall target loss function of the adversarial generative network model of this application is:

;

Among them, λ ₁ =1, λ ₂ =10, λ ₃ =10; L _GAN represents the adversarial loss, which mainly controls the authenticity of the picture output; L _FM represents the feature loss, which mainly controls the characteristics of the output picture in the discriminator/discriminator Consistent; L _VGG represents the perceptual loss, which mainly controls the consistency of features in the VGG convolutional network sub-model.

It should be noted that the specific acquisition processes of the adversarial loss LGAN , _the feature loss LFM and _{the perceptual loss LVGG} belong _to the existing technology in this field. Those skilled in the art can refer to the specific structure and training methods of the existing pix2pixHD model. The specific training process of the adversarial generative network model of this application will not be described again here.

It should be noted that the two loss functions, feature loss L _FM and perceptual loss L _VGG , come from the layer feature mapping of the discriminator and the pre-trained VGG-19 model respectively:

;

;

Among them, T represents the number of network layers of the discriminator D _i , T' represents the number of network layers of the VGG convolutional network submodel, f ^{( j )} _Di represents the j-th layer feature of the discriminator D _i , f ^{( j )} _VGG represents VGG The jth layer features of the convolutional network submodel. G refers to the generator (Generator), and D _i refers to the i -th discriminator (Discriminator). The generator is a neural network model; D is a neural network model similar to the discriminator in the existing pix2pixHD. I _g represents the original image collected (that is, the stereo camera image corresponding to the new perspective scene rendering image I _r ), which is used as the true value; C represents the discriminator D _i or the corresponding feature channel in the VGG convolution network submodel. Quantity, C _j is the number of feature channels of the jth layer network in the discriminator D _i or VGG convolutional network submodel.

It can be seen that the reconstruction and completion method proposed in this application adopts a data-driven automatic driving scene synthesis framework, which mainly includes the reconstruction and image completion network of the three-dimensional surface element model (such as the above-mentioned global surface element model). (i.e. adversarial generative network model). This reconstruction and completion method uses the stereo camera image obtained by shooting the autonomous driving scene by the stereo camera, and obtains the stereo camera image

The depth map and semantic map are optimized and filtered based on the semantic map to obtain an optimized depth map, which can effectively reconstruct the three-dimensional surface element model as described above.

In some embodiments, the three-dimensional surface element model uses an adaptive rendering mechanism to present new perspective images with reasonable quality.

The image completion network proposed in this application can use the proposed mask-image matching technology (ie, the above-mentioned mask image representation association mechanism MPC) to find the most approximate information from the scene context.

The CSD sub-module proposed in this application can enhance the content consistency of the synthetic image sequence (that is, the above-mentioned complete new perspective scene image sequence). Experimental results show that the reconstruction and completion method proposed in this application is about 5% to 73% higher than the existing state-of-the-art methods in terms of image quality, 45% to 162% higher in video quality, and 2D detection. In terms of performance, it is about 6%~24% higher under the indicator mAP@0.5, and 38%~41% higher in positioning accuracy. Among them, mAP@0.5 means mAP calculated under the IOU threshold of 0.5. mAP (mean Average Precision) is a very important measurement indicator in the field of target detection in machine learning and is used to measure the performance of target detection algorithms.

It can be seen that the reconstruction and completion method proposed in this application can be used to reconstruct static scene sequences of autonomous driving; wherein, the semantic information in the above semantic map is used to optimize the depth map to obtain an optimized depth map. The depth map is used to construct a global scene element model with adaptive rendering representation, and a new perspective scene rendering image is obtained from the global scene element model.

It can be seen that the reconstruction and completion method proposed in this application, in order to maintain the continuity of the spatial content of the generated image (that is, the complete new perspective scene image sequence), this application provides a mask image matching mechanism, which can be The collected scene data is matched with the contextual approximate information that is most similar to the rendered image (i.e., the above-mentioned new perspective scene rendering image).

It can be seen that the matched context approximate information is input into the normalization layer of the feature domain (ie, each of the aforementioned CSD sub-modules) through the CSD sub-module. Among them, the above "inputting the matched contextual approximate information into the normalization layer of the feature domain" refers to going through the convolution layer and normalization operation, which only represents an information flow process. The CSD sub-module can learn the prior similarity information (such as the above-mentioned modulation affine parameters) between the above-mentioned rendered image (i.e., the new perspective scene rendering image) and the reference image, and propagate it to the entire network of the adversarial generation network model.

The above are some instructions on the reconstruction and completion method of an autonomous driving scene. Please refer to Figure 8. Some embodiments of this application also disclose a system for reconstruction and completion of autonomous driving scenes, including:

Stereo camera 100 configured to acquire a stereo camera image of the autonomous driving scene;

A semantic depth generation module 200 configured to generate a depth map and a semantic map of the stereo camera image;

The semantic depth enhancement module 300 is configured to optimize the depth map using the semantic map to obtain an optimized depth map;

The surface element model building module 400 is configured to use the optimized depth map to establish a local surface element model of the current frame of the stereo camera image, and compare the local surface element model of the current frame with the previous one of the current frame. The conflict information that conflicts with the global scene element model corresponding to all image frames is eliminated, and the local plane element model of the current frame after eliminating the conflict information is used to fuse with the global scene element model until it is integrated with the automatic The global scene element model corresponding to all image frames of the driving scene is constructed; the global scene element model corresponding to all the image frames of the automatic driving scene is adaptively rendered to obtain the adaptively rendered global scene element. A model that obtains a new perspective scene image sequence corresponding to the autonomous driving scene from the adaptively rendered global element model; wherein the new perspective scene image sequence includes a plurality of new perspective scene rendering images;

The adversarial network completion module 500 is configured to obtain the rendered image pyramid of the new perspective scene rendering image, the reference image pyramid of the reference image corresponding to the new perspective scene rendering image, and the reference image pyramid corresponding to the new perspective scene rendering image. Mask image pyramid of the mask image, input the rendered image pyramid, the reference image pyramid and the mask image pyramid into a trained adversarial generative network model to use the adversarial generative network model to analyze the new perspective The missing information areas in the scene rendering image are completed to generate a complete new perspective scene image sequence;

Wherein, the complete new perspective scene image sequence includes multiple complete new perspective scene images; the reference image is the stereo camera image from an adjacent perspective of the new perspective scene rendering image, and the mask image is based on The new perspective scene is obtained by rendering the area with missing information in the image.

It should be noted that the role of the semantic depth enhancement module 300 is to optimize the depth of object boundaries in the depth map and eliminate dynamic objects (such as cars and people moving in the scene). That is, the semantic depth enhancement module 300 improves the depth of the object boundaries in the original depth map. Depth of different object boundaries. Among them, the specific content of the semantic depth enhancement module 300 to optimize the boundary depth according to the depth locality principle of the semantically consistent area has been discussed previously. This part is proposed by this application based on the problem of large depth error in the existing depth map. solution.

It should be noted that the specific execution process and technical effects of the reconstruction and completion system of this autonomous driving scene can be referred to the specific execution process and technical effects of the reconstruction and completion method of this autonomous driving scene. The specific execution process and technical effects will not be described here.

The above are some instructions on the reconstruction and completion system of an autonomous driving scene. Some embodiments of this application also disclose a computer-readable storage medium, including a program, which can be executed by a processor to implement the method as in any embodiment herein.

This document is described with reference to various exemplary embodiments. However, those skilled in the art will recognize that changes and modifications can be made to the exemplary embodiments without departing from the scope herein. For example, the various operational steps, as well as the components used to perform the operational steps, may be implemented in different ways (e.g., one or more steps may be eliminated, modified or incorporated into other steps).

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Additionally, as will be understood by those skilled in the art, the principles herein may be reflected in a computer program product on a computer-readable storage medium preloaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices (hard disk, floppy disk, etc.), optical storage devices (CD to ROM, DVD, Blu Ray disk, etc.), flash memory and/or the like . These computer program instructions may be loaded onto a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to form a machine, such that the instructions executed on the computer or other programmable data processing apparatus may generate a device that implements the specified functions. These computer program instructions may also be stored in a computer-readable memory, which may instruct a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory may form a Manufactured articles include devices that perform specified functions. Computer program instructions may also be loaded onto a computer or other programmable data processing device to perform a series of operating steps on the computer or other programmable device to produce a computer-implemented process such that the execution on the computer or other programmable device Instructions can provide steps for implementing a specified function.

Although the principles herein have been illustrated in various embodiments, many modifications of structure, arrangement, proportion, elements, materials and parts as are particularly suited to particular circumstances and operating requirements may be made without departing from the principles and scope of the disclosure. use. The above modifications and other changes or revisions are intended to be included within the scope of this document.

The foregoing detailed description has been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the disclosure. Accordingly, this disclosure is to be considered in an illustrative and not a restrictive sense, and all such modifications are to be included within its scope. Likewise, advantages, other advantages, and solutions to problems with respect to various embodiments have been described above. However, benefits, advantages, solutions to problems, and any elements that produce these, or make the solution more explicit, are not to be construed as critical, required, or essential. As used herein, the term "comprises" and any other variations thereof are intended to be non-exclusively inclusive such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also those not expressly listed or otherwise not part of the process , methods, systems, articles or other elements of equipment. Furthermore, the term "coupled" and any other variations thereof as used herein refers to physical connection, electrical connection, magnetic connection, optical connection, communication connection, functional connection and/or any other connection.

Those skilled in the art will recognize that many changes may be made in the details of the embodiments described above without departing from the basic principles of the invention. Therefore, the scope of the invention should be determined solely by the claims.

Claims (10)

1. A method for reconstructing and completing autonomous driving scenes, which is characterized by including: Obtain a stereo camera image of the automatic driving scene, and generate a depth map and a semantic map of the stereo camera image; wherein the stereo camera image is obtained by shooting the automatic driving scene by a stereo camera; the depth The graph is used to represent the depth information of the stereo camera image, and the semantic map is used to represent the semantic information of the stereo camera image; the semantic map is used to optimize the depth map to obtain an optimized depth map; Using the optimized depth map to establish a local block model of the current frame of the stereo camera image, the local block model of the current frame and the global block model corresponding to all image frames before the current frame Conflicting conflict information is eliminated, and the local surface element model of the current frame after eliminating the conflict information is used to fuse with the global scene element model until the global scene corresponding to all image frames of the autonomous driving scene is The metamodel construction is completed; Perform adaptive rendering on the global scene element model corresponding to all image frames of the autonomous driving scene to obtain the adaptively rendered global scene element model. From the adaptively rendered global scene element model Obtain a new perspective scene image sequence corresponding to the autonomous driving scene in the model; wherein the new perspective scene image sequence includes a plurality of new perspective scene rendering images; Obtain the rendered image pyramid of the new perspective scene rendering image, the reference image pyramid of the reference image corresponding to the new perspective scene rendering image, and the mask image pyramid of the mask image corresponding to the new perspective scene rendering image, and The rendered image pyramid, the reference image pyramid and the mask image pyramid input a trained adversarial generative network model to use the adversarial generative network model to supplement the areas with missing information in the new perspective scene rendering image. Completely generate a complete new perspective scene image sequence; Wherein, the complete new perspective scene image sequence includes multiple complete new perspective scene images; the reference image is the stereo camera image from an adjacent perspective of the new perspective scene rendering image, and the mask image is based on The new perspective scene is obtained by rendering the area with missing information in the image. 2. The reconstruction and completion method according to claim 1, wherein the adversarial generative network model includes a generator, and the generator includes multiple levels of CSD residual modules connected in series with each other; Wherein, the rendered image pyramid, the reference image pyramid and the mask image pyramid are input into a trained adversarial generative network model, so as to use the adversarial generative network model to detect defects in the new perspective scene rendering image. The information area is completed to generate a complete new perspective scene image sequence, including: The image results of each layer in the rendered image pyramid, the reference image pyramid and the mask image pyramid are respectively input to the CSD residual module of the first level in the generator to the end in order from the top layer to the bottom layer. The CSD residual module of the first level; The output of the CSD residual module of the previous stage is used as an input of the CSD residual module of the adjacent subsequent stage, and the output of the CSD residual module of the last stage is used as the complete new perspective scene image; Wherein, the network structure of the CSD residual module from the first level to the CSD residual module at the last level in the generator is the same; Wherein, the number of layers of the rendered image pyramid, the reference image pyramid and the mask image pyramid is the same as the number of the CSD residual modules. 3. The reconstruction and completion method according to claim 2, wherein the image results of each layer in the rendered image pyramid, the reference image pyramid and the mask image pyramid are in order from the top layer to the bottom layer. The CSD residual module of the first stage in the generator is input to the CSD residual module of the last stage respectively; the output of the CSD residual module of the previous stage is used as the output of the adjacent subsequent stage. An input of the CSD residual module, using the output of the last level CSD residual module as the complete new perspective scene image, includes: The image result of the top layer in the rendered image pyramid, the reference image pyramid and the mask image pyramid, and the new perspective scene rendering image corresponding to the scale of the image result of the top layer are input to the first level. Described CSD residual module; The image results of the remaining layers in the rendered image pyramid, the reference image pyramid and the mask image pyramid, except the top layer, are respectively input to each layer in the order from the next layer of the top layer to the bottom layer. The CSD residual module at the second level in the generator to the CSD residual module at the last level; use the output of the CSD residual module at the last level as the complete new perspective scene image. 4. The reconstruction and completion method according to claim 3, wherein the CSD residual module includes a first-level CSD sub-module, a second-level CSD sub-module and a third-level CSD sub-module; The data processing flow inside the CSD residual module is: The image results of the corresponding layers in the rendered image pyramid, the reference image pyramid and the mask image pyramid, as well as the feature maps input to the CSD residual module, are respectively input to the first-level CSD sub-module and The third-level CSD sub-module uses the output of the first-level CSD sub-module as the input of the first ReLU activation function, and performs a convolution operation on the output of the first ReLU activation function to obtain the first volume. product result, input the first convolution result into the second-level CSD sub-module, use the output of the second-level CSD sub-module as the input of the second ReLU activation function, and activate the second ReLU The output of the function is subjected to a convolution operation to obtain the second convolution result. The output of the third-level CSD sub-module is used as the input of the third ReLU activation function, and a convolution operation is performed on the output of the third ReLU activation function. To obtain a third convolution result, perform a unit addition operation on the second convolution result and the third convolution result to obtain the output of the CSD residual module; Wherein, the feature map of the CSD residual module input to the first level is the new perspective scene rendering image corresponding to the scale of the top-level image result, and the CSD residual module of the adjacent subsequent level is input The feature map is the output of the CSD residual module of the previous stage. 5. The reconstruction and completion method of claim 4, wherein the data of the first-level CSD sub-module, the second-level CSD sub-module and the third-level CSD sub-module The processing procedures are the same; Among them, the data processing flow of the CSD sub-module is: Similarity matching is performed on the input image results of the corresponding layer in the rendered image pyramid, the image results of the corresponding layer in the reference image pyramid, and the image results of the corresponding layer in the mask image pyramid to obtain the result of the rendering Contextual information about the image results of the corresponding layer in the image pyramid; Wherein, the context association information includes a similarity map and an associated image of the image result of the corresponding layer in the rendered image pyramid; the associated image is based on the image result of the corresponding layer in the reference image pyramid and the rendered image pyramid. The similarity map is generated from multiple regions where the image results of the corresponding layers are most similar. The similarity map is used to characterize the image results of the corresponding layer in the rendered image pyramid and the image results of the corresponding layer in the reference image pyramid. Correlation; Wherein, the context-related information is used to provide context matching information for the area with missing information; Perform a second convolution operation on the associated image and the similarity map to respectively obtain the spatial features of the associated image and the spatial features of the similarity map; Perform a second convolution operation on the image result of the corresponding layer in the rendered image pyramid to obtain the spatial characteristics of the new perspective scene rendering image of the corresponding layer; Perform element-by-element multiplication operations on the spatial features of the similarity map and the spatial features of the new perspective scene rendering image to obtain an element-by-element multiplication result; The second affine parameter is obtained by adding the element-wise multiplication result and the spatial feature of the associated image; wherein, the expression of the second affine parameter is: ; described/> is the element-wise multiplication result, the α _{c, h, w} are the spatial characteristics of the similarity map, and the /> is the spatial feature of the associated image; wherein, c, h and w are the number of characteristic channels, length and width of the image result of the corresponding layer respectively, _{and Ir} represents the new perspective scene rendering image, so The I _c' is the associated image of the image result of the corresponding layer; Normalize multiple feature maps input to the CSD residual module to obtain normalized parameters; The normalized result obtained by adding the normalized parameter and the second affine parameter is used as the output of the CSD sub-module. 6. The reconstruction and completion method according to claim 5, wherein the image result of the corresponding layer in the input rendered image pyramid, the image result of the corresponding layer in the reference image pyramid and the input Similarity matching is performed on the image results of the corresponding layer in the masked image pyramid to obtain contextual information related to the image results of the corresponding layer in the rendered image pyramid, including: Using a dividing frame to move in the image result of the corresponding layer in the rendered image pyramid in the first step to divide the image result of the corresponding layer in the rendered image pyramid into a plurality of rendering sub-blocks with a size of k×k, Utilize the dividing frame to move in the image result of the corresponding layer in the mask image pyramid in the first step to divide the image result of the corresponding layer in the mask image pyramid into a plurality of mask sub-sections with a size of k×k piece, The search domain corresponding to the rendering sub-block is determined in the image result of the corresponding layer in the reference image pyramid. The size of the search domain is k'×k', and the dividing frame is used to locate the search domain at the second step. Move in the search domain and divide the search domain into s×s reference sub-blocks with a size of k×k; where k, k' and s are all preset constants; Apply the mask sub-block corresponding to the rendering sub-block to the reference sub-block corresponding to the rendering sub-block to obtain a reference mask sub-block corresponding to the rendering sub-block; Calculate each similarity between the rendering sub-block and each reference mask sub-block in the search domain corresponding to the rendering sub-block, and use the maximum value among the similarities as the similarity Elements corresponding to the rendering sub-block in the figure; wherein the size of the search domain corresponding to the rendering sub-block is greater than the size of the rendering sub-block; The reference sub-block corresponding to each element in the similarity map is used as the best matching sub-block, and the associated image is generated using a plurality of the best-matching sub-blocks corresponding to the similarity map. . 7. The reconstruction and completion method according to claim 6, wherein the expression of the similarity graph is: ;wherein, simi ( i ) represents the i-th rendering sub-block in the image result of the corresponding layer in the rendering image pyramid/> and s×s reference mask sub-blocks in the search domain corresponding to the i-th rendering sub-block in the image result of the corresponding layer in the reference image pyramid/> The maximum value of each similarity between them, /> To obtain the cosine similarity operation, ⊙ represents element-wise multiplication, as described here , wherein the H and W are the height and width of the new perspective scene rendering image respectively, the n is the first frame; _{the Im} represents the mask image, and the I _c represents the The reference image, the j represents the reference sub-block in the search domain/> quantity, stated/> is the mask sub-block. 8. The reconstruction and completion method according to claim 5, characterized in that the expression of the normalized parameter is: , Wherein, the x ⁱ is the feature map of the CSD sub-module in the CSD residual module of the input level i, the x ⁱ ∈R ^N×C×H×W , and the C is the corresponding The number of feature channels; the μ _c ( xi ⁾ and σ _c ( xi ⁾ are respectively the mean of the N feature maps input to the CSD sub-module in the i-th level CSD residual module and standard deviation, the γ _c is the first affine parameter, and the γ _c ∈ R ^C ; the modulated affine parameters include the first affine parameter γ _c and the second affine parameter. 9. A system for reconstruction and completion of autonomous driving scenes, which is characterized by including: a stereo camera configured to acquire a stereo camera image of the autonomous driving scene; a semantic depth generation module configured to generate a depth map and a semantic map of the stereo camera image; A semantic depth enhancement module configured to optimize the depth map using the semantic map to obtain an optimized depth map; A surface element model building module configured to use the optimized depth map to establish a local surface element model of the current frame of the stereo camera image, and compare the local surface element model of the current frame with the previous one of the current frame. The conflict information that conflicts with the global scene element model corresponding to all image frames is eliminated, and the local segment model of the current frame after eliminating the conflict information is used to fuse with the global scene element model until it is integrated with the automatic driving The construction of the global scene element model corresponding to all image frames of the scene is completed; the global scene element model corresponding to all the image frames of the autonomous driving scene is adaptively rendered to obtain the adaptively rendered global scene element model. , obtain a new perspective scene image sequence corresponding to the autonomous driving scene from the adaptively rendered global element model; wherein the new perspective scene image sequence includes a plurality of new perspective scene rendering images; An adversarial network completion module configured to obtain a rendered image pyramid of the new perspective scene rendering image, a reference image pyramid of a reference image corresponding to the new perspective scene rendering image, and a mask corresponding to the new perspective scene rendering image. Mask image pyramid of the code image, input the rendered image pyramid, the reference image pyramid and the mask image pyramid into a trained adversarial generative network model to use the adversarial generative network model to analyze the new perspective scene The areas with missing information in the rendered image are completed to generate a complete new perspective scene image sequence; wherein the complete new perspective scene image sequence includes multiple complete new perspective scene images; the reference image is from the new perspective scene The stereo camera image is rendered from an adjacent perspective of the image, and the mask image is obtained based on the area of missing information in the new perspective scene rendering image. 10. A computer-readable storage medium, characterized by comprising a program that can be executed by a processor to implement the method according to any one of claims 1 to 8.