CN114331848A - Video image splicing method, device and equipment - Google Patents

Abstract

The application relates to a video image splicing method, a video image splicing device and video image splicing equipment. The video image splicing method comprises the following steps: obtaining the image proportion of the ROI and an original video image frame according to the vehicle-mounted radar ranging data and a first preset model; obtaining display position information of the ROI on the original video image frame according to the picture proportion and a second preset model; according to the picture proportion and the display position information, cutting the original video image frame to obtain a target ROI; and splicing the target ROI obtained by cutting into a target video image. The scheme provided by the application can improve the splicing effect of the video images.

Description

Video image stitching method, device and equipment

technical field

The present application relates to the technical field of automatic driving, and in particular, to a video image stitching method, device and device.

Background technique

With the continuous development of autonomous driving technology, more and more sensors with different functions are deployed in vehicles, for example, sensors such as cameras and/or various types of radars are deployed on the front, rear, left, and right sides of the vehicle. Among them, the radar is mainly used to measure the distance between the vehicle and the obstacle, and the camera is used to collect scene images around the vehicle. In remote driving, the vehicle can send the video images collected by the sensor to the remote cockpit, and the remote cockpit can stitch the video images.

In the related art, when splicing images, a traditional pure visual image splicing algorithm is adopted, and the problem of low real-time performance in the video image splicing method is solved by asynchronously updating the splicing model. However, the video image stitching processing method in the related art cannot solve the problem of depth of field under parallax, and the stitching effect is poor.

SUMMARY OF THE INVENTION

In order to solve or partially solve the problems existing in the related art, the present application provides a video image splicing method, device and device, which can improve the splicing effect of video images.

A first aspect of the present application provides a video image stitching method, including:

Obtain the ratio of the ROI of the region of interest to the original video image frame according to the vehicle radar ranging data and the first preset model;

Obtain the display position information of the ROI on the original video image frame according to the screen ratio and the second preset model;

According to the aspect ratio and the display position information, the target ROI is obtained by cropping the original video image frame;

The target ROI obtained from each crop is spliced into a target video image.

In one embodiment, the obtaining the ratio of the ROI of the region of interest to the original video image frame according to the vehicle-mounted radar ranging data and the first preset model includes: inputting the vehicle-mounted radar ranging data into a pre-trained shallow neural network. The model outputs the ratio of the ROI of the region of interest to the original video image frame;

The obtaining the display position information of the ROI on the original video image frame according to the picture scale and the second preset model includes: inputting the picture scale into a fitting model, and outputting the ROI in the original video frame. Display position information on video image frames.

In one embodiment, the shallow neural network model is obtained by training in the following manner:

The shallow neural network model is trained by using a training set to obtain the pre-trained shallow neural network model, wherein the training set includes annotated screen scale and training radar ranging data, and the marked screen scale is the mark The aspect ratio of the training ROI to the training video image frame.

In one embodiment, the training set is obtained as follows:

Select a set amount of data from the collected radar ranging data as the training radar ranging data;

According to the principle of picture continuity, the ROI that is continuous with the previous frame of the training video image frame is marked from the training video image frame, and the training ROI is obtained;

According to the ROI for labeling training and the video image frame for training, the labeling screen ratio is obtained;

The ratio of the radar ranging data for training and the marked picture is saved as a training set.

In one embodiment, the fitting model is obtained in the following manner:

The fitting model is obtained by fitting the labeling display position of the labeling training ROI on the training video image frame and the labeling screen ratio.

In one embodiment, the fitting model is obtained by fitting the labeling display position of the labeling training ROI on the training video image frame and the labeling screen ratio, including:

Inputting each of the marked screen ratios as known quantities into a target fitting equation with a target display position as an unknown quantity, and iteratively solving the target display position, wherein the target fitting equation includes a polynomial fitting coefficient;

When the deviation between the target display position and the labeling display position of the labeling training ROI on the training video image frame is less than a preset threshold, determine the corresponding polynomial fitting coefficient as the target fitting coefficient value, and the target fitting equation determined by the target fitting coefficient value is used as the fitting model.

A second aspect of the present application provides a video image splicing device, comprising:

The first output module is used to obtain the picture ratio of the region of interest ROI and the original video image frame according to the vehicle radar ranging data and the first preset model;

a second output module, configured to obtain the display position information of the ROI on the original video image frame according to the screen ratio and the second preset model;

a target area module, configured to obtain a target ROI by cropping the original video image frame according to the screen ratio obtained by the first output module and the display position information obtained by the second output module;

The splicing module is used for splicing the target ROI obtained by each cropping in the target area module into a target video image.

In one embodiment, the first output module inputs the vehicle-mounted radar ranging data into a pre-trained shallow neural network model, and outputs the screen ratio of the region of interest ROI and the original video image frame;

The second output module inputs the picture scale into a fitting model, and outputs the display position information of the ROI on the original video image frame.

In one embodiment, the apparatus further comprises:

A model training module is used to train the shallow neural network model by using a training set to obtain the pre-trained shallow neural network model, wherein the training set includes annotated screen ratio and training radar ranging data, so The annotated screen ratio is the screen ratio of the ROI for marking training and the video image frame for training.

A third aspect of the present application provides an electronic device, comprising:

processor; and

A memory having executable code stored thereon which, when executed by the processor, causes the processor to perform the method as described above.

A fourth aspect of the present application provides a computer-readable storage medium on which executable codes are stored, and when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the above method.

The technical solution provided by this application can include the following beneficial effects:

This application uses the vehicle radar ranging data as the data input. The data volume of the vehicle radar ranging data itself is relatively small, and the calculation amount of the input preset model is also relatively small. In addition, the vehicle radar ranging data can also reflect the video image. The depth of field is different, so the target ROI obtained according to the screen ratio of the region of interest ROI and the original video image frame and the display position information of the ROI on the original video image frame can make the subsequent stitched pictures more continuous, so that the stitching according to the target ROI can be made. The resulting target video image has a better picture display effect, thereby improving the video image splicing effect.

Further, the present application is to input the vehicle radar ranging data into a pre-trained shallow neural network model, output the picture ratio of the region of interest ROI and the original video image frame, and input the picture scale into the fitting model, and output the described picture ratio. Display position information of the ROI on the original video image frame. Since the number of layers of the shallow neural network model itself is small, compared with the deep neural network, the operation process of the shallow neural network model is much simpler, so the calculation speed is faster; and the fitting model belongs to the traditional mathematical model, It can also be done quickly when outputting the display position information of the ROI on the original video image frame. Therefore, the technical solution of the present application can meet the requirement of high real-time performance for video splicing.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

The above and other objects, features and advantages of the present application will become more apparent from the more detailed description of the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the exemplary embodiments of the present application. same parts.

1 is a schematic flowchart of a video image splicing method shown in an embodiment of the present application;

Fig. 2 is another schematic flowchart of the video image stitching method shown in the embodiment of the present application;

3 is another schematic flowchart of the video image stitching method shown in the embodiment of the present application;

4 is a schematic flowchart of training a model in the video image stitching method shown in the embodiment of the present application;

5 is a schematic flowchart of applying a model in the video image stitching method shown in the embodiment of the present application;

6 is a schematic structural diagram of a video image splicing device shown in an embodiment of the present application;

7 is another schematic structural diagram of the video image splicing device shown in the embodiment of the present application;

FIG. 8 is a schematic structural diagram of an electronic device shown in an embodiment of the present application.

Detailed ways

Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of this application to those skilled in the art.

The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, such information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information without departing from the scope of the present application. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature. In the description of the present application, "plurality" means two or more, unless otherwise expressly and specifically defined.

The video image stitching processing method in the related art cannot solve the problem of depth of field under parallax, and the stitching effect is poor. In view of the above problems, the embodiments of the present application provide a video image splicing method, which can improve the splicing effect of video images.

The technical solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of a video image stitching method according to an embodiment of the present application.

Referring to Figure 1, the method includes:

S101 , obtaining a screen ratio between the ROI of the region of interest and the original video image frame according to the vehicle-mounted radar ranging data and the first preset model.

In this step, the vehicle radar ranging data can be input into a pre-trained shallow neural network model, and the ratio of the ROI (Region Of Interest, region of interest) to the original video image frame is output.

Among them, the shallow neural network model is obtained by training in the following way: using the training set to train the shallow neural network model to obtain a pre-trained shallow neural network model, wherein the training set includes the scale of the labeled screen and the radar ranging data for training, The annotated frame ratio is the frame ratio of the ROI used for labeling training and the video image frame used for training.

Among them, the training set can be obtained in the following ways: select a set amount of data from the collected radar ranging data as the training radar ranging data; according to the principle of continuous picture, mark the training video image frame from the training video image frame and the previous frame training data. Use the continuous ROI of the video image frame to obtain the ROI for labeling training; compare the ROI for labeling training with the video image frame for training to obtain the proportion of labeling picture; save the radar ranging data for training and the proportion of labeling picture as a training set.

S102. Obtain the display position information of the ROI on the original video image frame according to the screen ratio and the second preset model.

In this step, the screen ratio can be input into the fitting model, and the display position information of the ROI on the original video image frame can be output.

Wherein, the fitting model is obtained in the following manner: The fitting model is obtained by fitting the labeling display position of the labeling training ROI on the training video image frame and the labeling screen scale.

For example, each annotated screen ratio can be input as a known quantity into a target fitting equation with the target display position as an unknown quantity, and the target display position can be solved iteratively, wherein the target fitting equation includes a polynomial fitting coefficient; When the deviation between the marked display positions of the ROI on the training video image frame is less than the preset threshold, the corresponding polynomial fitting coefficient is determined as the target fitting coefficient value, and the target is determined by the target fitting coefficient value. Fit the equation as the fitted model.

S103. According to the screen ratio and the display position information, the target ROI is obtained by cropping from the original video image frame.

Since the picture size of the original video image frame is known or fixed, when the ratio of the ROI to the original video image frame and the display position information of the ROI on the original video image frame are determined, it can be easily obtained by cropping from the original video image frame. The target ROR is the final ROI.

S104, splicing each cropped target ROI into a target video image.

After obtaining the target ROR, that is, the final ROI, when subsequent video image splicing is required, using the target ROI for splicing can make the images of the spliced video images more continuous and the splicing effect better.

It can be seen from this embodiment that the application uses the vehicle-mounted radar ranging data as the data input, the data volume of the vehicle-mounted radar ranging data itself is relatively small, and the calculation amount of the input preset model is also relatively small. In addition, the vehicle-mounted radar ranging data The data can also reflect the different depth of field of the video image, so the target ROI obtained according to the ratio of the ROI of the region of interest to the original video image frame and the display position information of the ROI on the original video image frame can make the subsequent stitched pictures more continuous. , so that the screen display effect of the target video image spliced according to the target ROI is better, thereby improving the splicing effect of the video image.

FIG. 2 is another schematic flowchart of the video image stitching method shown in the embodiment of the present application.

Referring to Figure 2, the method includes:

S201 , input the vehicle radar ranging data into a pre-trained shallow neural network model, and output the picture ratio between the ROI of the region of interest and the original video image frame.

The original video image frame is a video image frame of the vehicle driving environment collected by the vehicle-mounted camera.

In this embodiment of the present application, the vehicle-mounted radar may be an ultrasonic radar or a millimeter-wave radar, etc., which may be deployed around the vehicle, and the number may be deployed according to actual requirements, for example, 12 ultrasonic radars may be deployed. The vehicle-mounted radar ranging data mainly includes the distance information between the target or obstacle and the vehicle measured by the vehicle-mounted radar in real time. The frame's image depth of field. Therefore, the vehicle radar ranging data can also reflect the different depth of field of the video image.

A region of interest (Region Of Interest, ROI) is a region on an image that is focused on when analyzing and processing an image, and the region may be the position of a target or an obstacle on the image. In machine vision and image processing, the area that needs to be processed is outlined from the processed image in the form of boxes, circles, ellipses, irregular polygons, etc., which is called the region of interest. Typical shapes of ROIs can be rectangles or other shapes. When the ROI is a rectangle, the aspect ratio of the ROI to the original video image frame includes the ratio of the length of the ROI to the image length of the original video image frame and the ratio of the width of the ROI to the image width of the original video image frame.

The shallow neural network model can be a basic neural network model that only includes an input layer, a hidden layer and an output layer, and each layer uses a sigmoid (S-shaped function) as an activation function. The sigmoid function is used for the output of neurons in the hidden layer. The value range is (0, 1). It can map a real number to the interval of (0, 1) and can be used for binary classification. The effect is better when the feature difference is more complex or the difference is not particularly large. Since the number of layers of the shallow neural network model itself is small, compared with the deep neural network, the operation process of the shallow neural network model is much simpler. Therefore, the calculation speed is faster and can better meet the high real-time requirements.

The present application can use the training set to train the shallow neural network model to obtain a pre-trained shallow neural network model. Among them, the training set includes the scale of the labeled screen and the radar ranging data for training. The scale of the labeled picture is the proportion of the labeled training ROI and the training video image frame, and the labeled display position is the labeled training ROI in the training video image frame. The label on the display shows the location. Among them, the training set can be obtained in the following ways: select a set amount of data from the collected radar ranging data as the training radar ranging data; according to the principle of continuous picture, mark the training video image frame from the training video image frame and the previous frame training data. Use the continuous ROI of the video image frame to obtain the ROI for labeling training; compare the ROI for labeling training with the video image frame for training to obtain the proportion of labeling picture; save the radar ranging data for training and the proportion of labeling picture as a training set.

S202. Input the aspect ratio of the ROI and the original video image frame into the fitting model, and output the display position information of the ROI on the original video image frame.

In this embodiment of the present application, the function of the fitting model is that when the aspect ratio of the ROI and the original video image frame is input, the display position information of the ROI on the original video image frame can be output.

Similar to the pre-training of the shallow neural network model before application, the fitting model here can also be a mathematical model obtained by fitting. The present application can perform fitting processing on the labeling display position of the labeling training ROI on the training video image frame and the labeling screen ratio to obtain a fitting model.

For example, input the scale of each annotated screen as a known quantity into a target fitting equation with the target display position as the unknown quantity, and iteratively solve the target display position, where the target fitting equation includes polynomial fitting coefficients; When the deviation between the marked display positions of the ROI on the training video image frame is less than the preset threshold, the corresponding polynomial fitting coefficient is determined as the target fitting coefficient value, and the target fitting coefficient value is determined by the value of the target fitting coefficient. The target fit equation is used as the fitted model.

S203 , according to the aspect ratio of the ROI and the original video image frame and the display position information of the ROI on the original video image frame, crop the target ROI from the original video image frame.

Since the picture size of the original video image frame is known or fixed, when the ratio of the ROI to the original video image frame and the display position information of the ROI on the original video image frame are determined, it can be easily obtained by cropping from the original video image frame. The target ROR is the final ROI.

S204, splicing the target ROIs obtained by each cropping into target video image frames.

The target ROI obtained by the above process may be the original video image frames from the same camera or different cameras (for example, the front camera and the left camera or the right camera). When splicing these target ROIs, it can be A camera, such as the original video image frame collected by the camera in front of the car, or the target ROI of the video image frame with better quality as the benchmark.

The splicing of each cropped target ROI into a target video image frame can be realized by using an existing splicing processing method, for example, splicing is performed according to reference factors such as the pose or shooting time corresponding to the video image frame, which is not described in this application. limited.

It can be seen from this embodiment that the application is to input the vehicle radar ranging data into a pre-trained shallow neural network model, output the screen ratio of the region of interest ROI and the original video image frame, and input the screen ratio into the fitting model, Output the display position information of the ROI on the original video image frame. Since the number of layers of the shallow neural network model itself is small, compared with the deep neural network, the operation process of the shallow neural network model is much simpler, so the calculation speed is faster; and the fitting model belongs to the traditional mathematical model, It can also be done quickly when outputting the display position information of the ROI on the original video image frame. Therefore, the technical solution of the present application can meet the requirement of high real-time performance for video splicing.

FIG. 3 is another schematic flowchart of the video image stitching method shown in the embodiment of the present application. FIG. 3 describes the solution of the present application in more detail with respect to FIGS. 1 and 2 .

Referring to Figure 3, the method includes:

S301 , collecting vehicle-mounted radar ranging data and video image frames of a camera and performing preprocessing.

In this application, when an object around the driving vehicle enters the effective range of the ultrasonic radar, the video image frame data and the vehicle radar ranging data of all the current cameras can be recorded every set time, such as 5s, and saved.

It should be noted that, in order to facilitate the processing of training the shallow neural network model and further reduce the amount of calculation, in this embodiment of the present application, the vehicle radar ranging data may be preprocessed. The preprocessing includes, for example, data cleaning (for example, removing dirty point data, that is, ranging data that obviously does not meet the requirements) and data normalization (normalization), etc.; The language is more intuitive and convenient.

S302 , pre-train the shallow neural network model to obtain a trained shallow neural network model, and perform fitting processing on the fitting model to obtain the processed fitting model.

The processing process of this step can be referred to as shown in FIG. 4 , which is a schematic flowchart of training a model in the video image stitching method shown in the embodiment of the present application.

The present application can use the training set to train the shallow neural network model to obtain a pre-trained shallow neural network model. The training set can be obtained in the following ways: select a set amount of data from the collected radar ranging data as the training radar ranging data; according to the principle of continuous picture, mark the training video image frame from the training video image frame with the previous frame of the training video. The continuous ROI of the image frame is obtained to obtain the ROI for labeling training; according to the comparison between the ROI for labeling training and the video image frame for training, the proportion of labeling picture is obtained; the radar ranging data for training and the proportion of labeling picture are saved as the training set.

Specifically, the following describes the process of acquiring the training set through steps S1 to S4.

Step S1: Select a set amount of data from the collected radar ranging data as training radar ranging data.

From the collected radar ranging data, the first set amount of data can be selected as the training radar ranging data, the second set amount of data can be selected as the test radar ranging data, and the third set amount of data can be selected as the verification data. Radar ranging data. For example, from the collected radar ranging data, 50% are randomly selected as training radar ranging data, and the rest are used for testing or validating the shallow neural network model after training. The selected training radar ranging data may be, for example, radar ranging data of six ultrasonic radars in the forward direction of the vehicle.

The radar ranging data measures the distance information between the target or obstacle and the vehicle in real time, and the distance information can correspond to the image depth of the vehicle driving environment video image frame captured by the vehicle-mounted camera at the same time as the vehicle-mounted radar collects the ranging data.

Step S2: According to the principle of picture continuity, mark the ROI that is continuous with the previous frame of the training video image frame from the training video image frame, so as to obtain the labeling training ROI.

Specifically, you can select an area in the training video image frame to stretch or shrink, and stop the operation when it is found to be continuous with the previous frame of the training video image frame, then the area obtained at this time is the ROI for labeling training. The length of the picture for labeling the ROI for training is the labeling length, and the width of the picture is the labeling width. For example, if it is judged that there is no obvious jump between the two pixels, it can be determined that the picture is continuous.

Step S3: According to the comparison between the ROI for labeling training and the video image frame for training, the scale of the labeling screen is obtained.

In this step, the labeled training ROI is compared with the training video image frame to obtain the labeled screen ratio. When the labeled training ROI is a rectangle, the labeled screen ratio includes the ratio of the labeled length of the ROI to the picture length of the original video image frame and the ratio of the labeled width of the ROI to the picture width of the original video image frame.

Step S4: Save the radar ranging data for training and the ratio of the labeled picture as a training set.

Save the training radar ranging data and the scale of the labeled screen as a training set, and then input the training radar ranging data and the corresponding scale of the labeled screen into the training shallow neural network model to complete the training of the shallow neural network model.

The implementation manner of performing fitting processing on the fitting model to obtain the processed fitting model may include:

Input the scale of each annotated screen as a known quantity into a target fitting equation with the target display position as an unknown quantity, and iteratively solve the target display position, wherein the target fitting equation includes a polynomial fitting coefficient;

When the deviation between the target display position and the marked training ROI on the training video image frame is less than the preset threshold, determine the corresponding polynomial fitting coefficient as the target fitting coefficient, and use the The target fitting equation determined by the value of the target fitting coefficient is used as the fitting model. The preset threshold may be, for example, but not limited to, 0.05.

In the above embodiment, the marked display position is the actual value in the fitting process, the target display position is the predicted value in the fitting process, and the deviation can be the mean square error (Mean Square Error, MSE) between the target display position and the marked display position. ) or root mean square error (Root Mean Square Error, RMSE), the target fitting equation can be a linear equation or a nonlinear equation. In addition, it should be noted that when the deviation between the target display position and the label display position is smaller than the preset threshold, it can be considered that the deviation between the target display position and the label display position is the smallest at this time.

For example, assuming that the scale of the annotation screen, that is, the scale data (k), and the display position of the annotation, that is, the position data (x, y), are linearly correlated, the target fitting equation can be exemplified as follows, but is not limited to this:

x=a+bk+ck^2+dk^3+ek^4+fk^5+gk^6

y=h+ik+jk^2+lk^3+mk^4+nk^5+pk^6

Among them, a, b, c, d, e, f, g in the first polynomial equation and h, i, j, l, m, n, p in the second polynomial equation are polynomial fitting coefficients.

The so-called fitting is to connect a series of points on the plane with a smooth curve. Because this curve has an infinite number of possibilities, there are various fitting methods. The fitted curve can generally be represented by a function, and there are different fitting names depending on the function. Common fitting methods include, for example, the least squares curve fitting method, etc. In MATLAB (a mathematical software) tool, the polyfit function can also be used to fit polynomials. The polyfit function is based on the least squares method. In the above polynomial equations, each formula can be solved as long as there are seven or more sets of nonlinear data, that is, by using the polyfit function in MATLAB for fitting processing, the position data (x, y) and the true value (labeled) can be obtained. The value of the corresponding polynomial fitting coefficient when the display position) is closest, then the corresponding polynomial fitting coefficient value at this time can be used as the target fitting coefficient value, and the target determined by the target fitting coefficient value Fit the equation as the fitted model.

It should be noted that the specific process of using the polyfit function to fit the polynomial in MATLAB can be implemented according to the prior art, which is not limited in this application. In addition, the polynomials of the above equations are illustrated by taking the heptapomials as an example, but are not limited thereto.

S303 , input the vehicle radar ranging data into the pre-trained shallow neural network model, and output the picture ratio of the ROI of the region of interest and the original video image frame.

The shallow neural network model can be a basic neural network model that only includes an input layer, a hidden layer and an output layer, and each layer uses a sigmoid (S-shaped function) as an activation function. Since the number of layers of the shallow neural network model itself is small, compared with the deep neural network, the operation process of the shallow neural network model is much simpler. Therefore, the calculation speed is faster and can better meet the high real-time requirements.

In this step, the vehicle radar ranging data can be input into the pre-trained shallow neural network model for operation processing, and the picture ratio of the ROI of the region of interest and the original video image frame can be output. For the description of this step, reference may be made to the description in step S201, and details are not repeated here.

S304. Input the picture ratio of the ROI and the original video image frame into the fitting model, and output the display position information of the ROI on the original video image frame.

In this embodiment of the present application, the function of the fitting model is that when the aspect ratio of the ROI and the original video image frame is input, the display position information of the ROI on the original video image frame can be output.

In this step, the aspect ratio of the ROI and the original video image frame is input into the fitting model for fitting operation, and the display position information of the ROI on the original video image frame can be output.

S305 , according to the aspect ratio of the ROI and the original video image frame and the display position information of the ROI on the original video image frame, crop the target ROI from the original video image frame.

Since the picture size of the original video image frame is known or fixed, when the ratio of the ROI to the original video image frame and the display position information of the ROI on the original video image frame are determined, it can be easily obtained by cropping from the original video image frame. The target ROR is the final ROI.

The process of the above steps S303 to S305 can also be referred to as shown in FIG. 5 . FIG. 5 is a schematic flowchart of applying a model in the video image stitching method shown in the embodiment of the present application.

S306, splicing the target ROIs obtained by each cropping into target video image frames.

The target ROI obtained by the above process may be the original video image frames from the same camera or different cameras (for example, the front camera and the left camera or the right camera). When splicing these target ROIs, it can be A camera, such as the original video image frame collected by the camera in front of the car, or the target ROI of the video image frame with better quality as the benchmark.

The splicing of each cropped target ROI into a target video image frame can be realized by using an existing splicing processing method, for example, splicing is performed according to reference factors such as the pose or shooting time corresponding to the video image frame, which is not described in this application. limited.

It can be seen from this embodiment that the application uses vehicle-mounted radar ranging data as the data input, the data volume of the vehicle-mounted radar ranging data itself is relatively small, the calculation amount of the input preset model is also relatively small, and the vehicle-mounted radar ranging data is also relatively small. It can reflect that the depth of field of the video image is different; in addition, the application also adopts the operation of the shallow neural network model, because the number of layers of the shallow neural network model itself is small, compared with the deep neural network, the The operation process is much simpler and the operation speed is fast. Therefore, the technical solution of the present application can meet the high real-time requirement of video splicing, and can also make the display effect of the target video image spliced according to the target ROI better, thereby improving the splicing effect of the video image.

Corresponding to the foregoing application function implementation method embodiments, the present application further provides a video image splicing device, an electronic device, and corresponding embodiments.

FIG. 6 is a schematic structural diagram of a video image splicing apparatus shown in an embodiment of the present application.

Referring to FIG. 6 , a video image splicing device 60 provided by the present application includes: a first output module 601 , a second output module 602 , a target area module 603 , and a splicing module 604 .

The first output module 601 is configured to obtain the picture ratio of the region of interest ROI and the original video image frame according to the vehicle radar ranging data and the first preset model. Wherein, the first preset model may be a shallow neural network model, wherein the shallow neural network model is obtained by training in the following manner: using a training set to train the shallow neural network model to obtain a pre-trained shallow neural network model, The training set includes annotated screen ratio and training radar ranging data, and the marked screen ratio is the screen ratio of the ROI for marking training and the video image frame for training.

The second output module 602 is configured to obtain the display position information of the ROI on the original video image frame according to the screen ratio and the second preset model. Wherein, the second preset model may be a fitting model. The fitting model can be obtained in the following manner: the fitting model is obtained by fitting the labeling display position of the labeling training ROI on the training video image frame and the labeling screen ratio.

The target area module 603 is configured to obtain the target ROI by cropping the original video image frame according to the screen ratio obtained by the first output module 601 and the display position information obtained by the second output module 602 . Since the picture size of the original video image frame is known or fixed, when the aspect ratio of the ROI to the original video image frame and the display position information of the ROI on the original video image frame are determined, the target area module 603 can easily extract the image from the original video image. The image frame is cropped to obtain the target ROR, which is the final ROI.

The splicing module 604 is used for splicing the target ROI obtained by each cropping in the target region module 603 into a target video image. Using the target ROI for stitching can make the picture of the stitched video images more continuous and the stitching effect is better.

It can be seen from this embodiment that the video image splicing device provided by the present application uses the vehicle radar ranging data as the data input, the data volume of the vehicle radar ranging data itself is relatively small, and the calculation amount of the input preset model is also relatively small In addition, the vehicle radar ranging data can also reflect the different depth of field of the video image, so the target ROI obtained according to the ratio of the ROI of the region of interest to the original video image frame and the display position information of the ROI on the original video image frame can make The subsequent stitched pictures are more continuous, so that the picture display effect of the target video image stitched according to the target ROI is better, thereby improving the stitching effect of the video images.

FIG. 7 is another schematic structural diagram of a video image splicing apparatus shown in an embodiment of the present application.

Referring to FIG. 7 , a video image stitching apparatus 60 provided by the present application includes: a first output module 601 , a second output module 602 , a target area module 603 , a stitching module 604 , a model training module 605 , and a data collection module 606 .

For the functions of the first output module 601, the second output module 602, the target area module 603, and the splicing module 604, reference may be made to the description in FIG. 6 .

Further, the first output module 601 can input the vehicle radar ranging data into the pre-trained shallow neural network model, and output the picture ratio of the region of interest ROI and the original video image frame.

The second output module 602 can input the screen ratio into the fitting model, and output the display position information of the ROI on the original video image frame.

The model training module 605 is used to train the shallow neural network model by using the training set to obtain a pre-trained shallow neural network model, wherein the training set includes the marked screen ratio and the radar ranging data for training, and the marked screen ratio is marked training The aspect ratio of the ROI and the video image frame used for training. Among them, the training set can be obtained in the following ways: select a set amount of data from the collected radar ranging data as the training radar ranging data; according to the principle of continuous picture, mark the training video image frame from the training video image frame and the previous frame training data. Use the continuous ROI of the video image frame to obtain the ROI for labeling training; compare the ROI for labeling training with the video image frame for training to obtain the proportion of labeling picture; save the radar ranging data for training and the proportion of labeling picture as a training set.

The model training module 605 may also perform a fitting process on the marked display position of the marked training ROI on the training video image frame and the marked screen scale to obtain a fitted model. For example, input the scale of each annotated screen as a known quantity and input the target fitting equation with the target display position as the unknown quantity, and iteratively solve the target display position, where the target fitting equation includes polynomial fitting coefficients; When the deviation between the marked display positions of the ROI on the training video image frame is less than the preset threshold, the corresponding polynomial fitting coefficient is determined as the target fitting coefficient value, and the target fitting coefficient determined by the target fitting coefficient value is determined. The fitting equation is used as the fitting model.

The data collection module 606 is used to collect vehicle radar ranging data and video image frames of the camera and perform preprocessing. In this application, when an object around the driving vehicle enters the effective range of the ultrasonic radar, the video image frame data and the vehicle radar ranging data of all the current cameras can be recorded every set time, such as 5s, and saved. In order to facilitate the training of the shallow neural network model and further reduce the amount of computation, the data collection module 606 may also preprocess the vehicle radar ranging data. The preprocessing includes, for example, data cleaning (for example, removing dirty point data, that is, ranging data that obviously does not meet the requirements) and data normalization (normalization), etc.; The language is more intuitive and convenient.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 8 is a schematic structural diagram of an electronic device shown in an embodiment of the present application.

Referring to FIG. 8 , an electronic device 800 includes a memory 810 and a processor 820 .

The processor 820 may be a central processing unit (Central Processing Unit, CPU), and may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-available processors Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Memory 810 may include various types of storage units, such as system memory, read only memory (ROM), and persistent storage. The ROM may store static data or instructions required by the processor 820 or other modules of the computer. Persistent storage devices may be readable and writable storage devices. Permanent storage may be a non-volatile storage device that does not lose stored instructions and data even if the computer is powered off. In some embodiments, persistent storage devices employ mass storage devices (eg, magnetic or optical disks, flash memory) as persistent storage devices. In other embodiments, persistent storage may be a removable storage device (eg, a floppy disk, an optical drive). System memory can be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. System memory can store some or all of the instructions and data that the processor needs at runtime. Additionally, memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (eg, DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic and/or optical disks may also be employed. In some implementations, memory 810 may include a removable storage device that is readable and/or writable, such as a compact disc (CD), a read-only digital versatile disc (eg, DVD-ROM, dual-layer DVD-ROM), Read-only Blu-ray Disc, Ultra-Density Disc, Flash Card (eg SD Card, Min SD Card, Micro-SD Card, etc.), Magnetic Floppy Disk, etc. Computer readable storage media do not contain carrier waves and transient electronic signals transmitted over wireless or wire.

Executable codes are stored on the memory 810, and when the executable codes are processed by the processor 820, the processor 820 can be caused to execute some or all of the above-mentioned methods.

Furthermore, the method according to the present application can also be implemented as a computer program or computer program product comprising computer program code instructions for performing some or all of the steps in the above method of the present application.

Alternatively, the present application can also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) on which executable codes (or computer programs or computer instruction codes) are stored, When the executable code (or computer program or computer instruction code) is executed by the processor of the electronic device (or server, etc.), the processor is caused to perform some or all of the steps of the above method according to the present application.

Various embodiments of the present application have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (11)

1. a video image splicing method, is characterized in that, comprises: Obtain the ratio of the ROI of the region of interest to the original video image frame according to the vehicle radar ranging data and the first preset model; Obtain the display position information of the ROI on the original video image frame according to the screen ratio and the second preset model; According to the aspect ratio and the display position information, the target ROI is obtained by cropping the original video image frame; The target ROI obtained from each crop is spliced into a target video image. 2. method according to claim 1, is characterized in that: The obtaining the ratio of the ROI of the region of interest to the original video image frame according to the vehicle-mounted radar ranging data and the first preset model includes: inputting the vehicle-mounted radar ranging data into a pre-trained shallow neural network model, and outputting the region of interest The aspect ratio of the ROI and the original video image frame; The obtaining the display position information of the ROI on the original video image frame according to the picture scale and the second preset model includes: inputting the picture scale into a fitting model, and outputting the ROI in the original video frame. Display position information on video image frames. 3. The method according to claim 2, wherein the shallow neural network model is obtained by training in the following manner: The shallow neural network model is trained by using a training set to obtain the pre-trained shallow neural network model, wherein the training set includes annotated screen scale and training radar ranging data, and the marked screen scale is the mark The aspect ratio of the training ROI to the training video image frame. 4. The method according to claim 3, wherein the training set is obtained in the following manner: Select a set amount of data from the collected radar ranging data as the training radar ranging data; According to the principle of picture continuity, the ROI that is continuous with the previous frame of the training video image frame is marked from the training video image frame, and the training ROI is obtained; According to the ROI for labeling training and the video image frame for training, the labeling screen ratio is obtained; The ratio of the radar ranging data for training and the marked picture is saved as a training set. 5. The method according to claim 2, wherein the fitting model is obtained in the following manner: The fitting model is obtained by fitting the labeling display position of the labeling training ROI on the training video image frame and the labeling screen ratio. 6. The method according to claim 5, characterized in that, performing fitting processing on the labeling display position of the labeling training ROI on the training video image frame and the labeling screen ratio, to obtain the result. described fitting model, including: Inputting each of the marked screen ratios as known quantities into a target fitting equation with a target display position as an unknown quantity, and iteratively solving the target display position, wherein the target fitting equation includes a polynomial fitting coefficient; When the deviation between the target display position and the labeling display position of the labeling training ROI on the training video image frame is less than a preset threshold, determine the corresponding polynomial fitting coefficient as the target fitting coefficient value, and the target fitting equation determined by the target fitting coefficient value is used as the fitting model. 7. A video image splicing device is characterized in that, comprising: The first output module is used to obtain the picture ratio of the region of interest ROI and the original video image frame according to the vehicle radar ranging data and the first preset model; a second output module, configured to obtain the display position information of the ROI on the original video image frame according to the screen ratio and the second preset model; a target area module, configured to obtain a target ROI by cropping the original video image frame according to the screen ratio obtained by the first output module and the display position information obtained by the second output module; The splicing module is used for splicing the target ROI obtained by each cropping in the target area module into a target video image. 8. The device according to claim 7, wherein: The first output module inputs the vehicle-mounted radar ranging data into the pre-trained shallow neural network model, and outputs the picture ratio of the region of interest ROI and the original video image frame; The second output module inputs the picture scale into a fitting model, and outputs the display position information of the ROI on the original video image frame. 9. The apparatus of claim 8, wherein the apparatus further comprises: A model training module is used to train the shallow neural network model by using a training set to obtain the pre-trained shallow neural network model, wherein the training set includes annotated screen ratio and training radar ranging data, so The annotated screen ratio is the screen ratio of the ROI for marking training and the video image frame for training. 10. An electronic device, comprising: processor; and A memory having executable code stored thereon which, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 6. 11. A computer-readable storage medium on which executable codes are stored, and when the executable codes are executed by a processor of an electronic device, the processor is caused to execute as claimed in any one of claims 1 to 6. method described.

Images