CN112862671B - Video image editing and repairing method, device and storage medium - Google Patents

Abstract

A video image editing method comprising: selecting a frame of target frame image and a frame of auxiliary frame image in an input video, and respectively pairing to obtain a plurality of pairing combinations, wherein each pairing combination comprises the target frame image and the frame of auxiliary frame image; extracting a pair of optical flows for each pairing combination respectively, including: reverse optical flow from the target frame image to the auxiliary frame image in the paired set, forward optical flow from the auxiliary frame image to the target frame image in the paired set; obtaining consistency marks of the images according to consistency judgment results of the reverse optical flow and the forward optical flow in each pair of optical flows; obtaining a segmentation mark corresponding to each instance in the target frame image; and respectively determining the coincidence degree of the segmentation marks corresponding to each instance and the coincidence marks, and combining the segmentation marks with the coincidence degrees higher than the preset condition to obtain combined marks, wherein the combined marks are used for marking the result of removing the region to be removed in the target frame image.

Description

Video image editing, repairing method, device and storage medium

Technical Field

This article relates to but is not limited to the fields of computer vision and digital image processing, and in particular to methods, devices and storage media for video image editing and restoration.

Background technique

With the continuous development of Internet technology and the widespread popularity of smart phones, mobile photography has become an important part of people's lives, and image/video editing has become an increasingly important issue in the field of image processing. In image/video editing technology, image/video restoration (also known as object removal) is particularly difficult. Its purpose is to remove non-subject objects in images/videos and fill in occluded areas. In some restoration methods, marking the areas that need to be removed needs to be done manually, which leads to inconvenience in use.

Summary of the invention

The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

The embodiment of the present application provides a video image editing method, which can solve the problem of having to rely on manual marking of areas that need to be removed during image/video repair.

The present application provides a video image editing method, including:

Selecting a target frame image and multiple auxiliary frame images from the input video;

Pairing the target frame image with multiple auxiliary frame images respectively to obtain multiple pairing combinations, each pairing combination including the target frame image and one auxiliary frame image;

For each pairing combination, a pair of optical flows is extracted respectively, including: a reverse optical flow and a forward optical flow; the reverse optical flow is an optical flow from the target frame image to the auxiliary frame image in the pairing combination, and the forward optical flow is an optical flow from the auxiliary frame image in the pairing combination to the target frame image;

According to the consistency judgment result of the reverse optical flow and the forward optical flow in each pair of optical flows, the consistency mark of the image is obtained;

Obtain the segmentation mark corresponding to each instance in the target frame image;

Determine the degree of overlap between the segmentation mark and the consistency mark corresponding to each instance respectively, and use the segmentation mark with a degree of overlap higher than a preset condition to combine with the consistency mark to obtain a combined mark, and the combined mark is used to identify the result after removing the area to be removed in the target frame image.

The embodiment of the present application also provides a video image repair method, including: the above-mentioned video image editing method, the method further comprising:

Each auxiliary frame image is propagated according to the forward optical flow, and the pixels required in the propagation result are selected according to the combined mark, and the average of all the selected pixels at the corresponding position is taken as the filling content to fill the target frame image to obtain the repaired image.

An embodiment of the present application also provides a video image editing device, including a memory and a processor, wherein the memory is used to store a program for video image editing; the processor is used to read the program for video image editing and execute the video image editing method as described above.

An embodiment of the present application also provides a video image repair device, including a memory and a processor, wherein the memory is used to store a program for performing video image repair; the processor is used to read the program for performing video image repair and execute the video image repair method as described above.

The embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the above-mentioned video image editing method or video image repair method.

The embodiment of the present application utilizes the time series relationship in the video, and can automatically identify non-subject objects that need to be removed in the input video and automatically remove areas that need to be removed in the image, thereby achieving automatic repair of the image/video with good repair effect, fast processing speed, manpower saving, and easy use.

Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or understood by implementing the embodiments of the present application. Other advantages of the embodiments of the present application can be realized and obtained by the schemes described in the description and the drawings.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide an understanding of the technical solutions of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solutions of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

FIG1 is a flow chart of a video image editing method according to an embodiment of the present application;

FIG2 is a schematic diagram of a video image editing or video image repairing device in an embodiment of the present application;

FIG3 is a flow chart of a video image editing and repairing method in an example of the present application.

Detailed ways

The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and/or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

In addition, when describing representative embodiments, the specification may have presented the method and/or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and/or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

When performing image/video restoration, it can be divided into two categories: image restoration and video restoration according to the different restoration objects:

The commonly used method in image restoration is the patch-based method: it searches for the pixel block in the image that is most similar to the edge of the damaged area, and then fills the pixel values in the similar pixel block into the damaged area through a series of transformations. This method generally completes the repair through continuous iterative optimization, so the speed is very slow. After deep learning methods were applied to this field, the Encoder-Decoder network structure was widely used. The network takes the original image as input and directly outputs the repaired image, which has greatly improved the speed. However, the inventors of this application have keenly discovered in practice that it is impossible to accurately restore the area that needs to be repaired using only image information.

The commonly used method in video restoration is also based on the patch method. Initially, people simply used the image restoration algorithm independently for each frame in the video, which did not take advantage of the temporal series relationship in the video. A more common approach is to find similar space-time pixel blocks in the video and then fill these pixel blocks into the blank area. However, similar to the image restoration method, this optimization method also has the disadvantage of being time-consuming. Another commonly used method is the optical flow-based method, which calculates the optical flow between two adjacent frames in the video and uses the optical flow information to transfer the intact part of the video directly to the damaged hole. If a deep network is used to estimate the optical flow, this method can achieve better results and higher speed. However, this method cannot accurately restore the area that needs to be repaired.

Furthermore, the inventors of the present application have also found in practice that the above-mentioned repair methods are all based on the assumption that the areas to be repaired in the image/video have been marked, but marking the areas to be removed still needs to be done manually, which is not feasible in practical applications.

As shown in FIG1 , the embodiment of the present application provides a video image editing method, comprising:

S100: selecting a target frame image and multiple auxiliary frame images in an input video;

S101: Pairing the target frame image with multiple auxiliary frame images respectively and extracting optical flows;

This step includes: pairing the target frame image with multiple auxiliary frame images respectively to obtain multiple pairing combinations, each pairing combination includes the target frame image and one auxiliary frame image;

For each pairing combination, a pair of optical flows is extracted respectively, including: a reverse optical flow and a forward optical flow; the reverse optical flow is the optical flow from the target frame image to the auxiliary frame image in the pairing combination, and the forward optical flow is the optical flow from the auxiliary frame image in the pairing combination to the target frame image;

S102: Obtaining a consistency mark of the image based on the consistency judgment result of each pair of optical flows;

According to the consistency judgment result of the reverse optical flow and the forward optical flow in each pair of optical flows, the consistency mark of the image is obtained;

S103: Obtaining a segmentation mark corresponding to each instance in the target frame image;

S104: Obtain a combined tag using the segmentation tag and the consistency tag; this step includes:

The degree of overlap between the segmentation mark and the consistency mark corresponding to each instance is determined respectively, and the segmentation mark with an overlap higher than a preset condition is combined with the consistency mark to obtain a combined mark. The combined mark is used to identify the result after removing the area to be removed in the target frame image.

In step S100, the target frame image is selected as the main operation object, that is, the target frame image is the object that needs to be repaired, and the subsequent operations will be performed based on the target frame image; the selected auxiliary frame image is used to compare with the target frame image and provide certain information. Based on the information provided by the auxiliary frame image, the target frame image can be edited and repaired. The target frame image can be selected arbitrarily, such as the first frame image, the last frame image, and any frame image in the middle of the video. The auxiliary frame images should select images other than the target frame image, and should be distributed as evenly as possible in the entire video sequence, so as to capture the entire video content as much as possible for reference, and provide more comprehensive information for editing the target frame image. When video repair is required, each frame of the video that needs to be repaired can be selected as a target frame image for repair.

The embodiment of the present application does not limit the execution order of the above steps S101 and S103, and can be selected as needed.

In this embodiment, an instance corresponds to an entity on the image screen, which can be each different object appearing in the image. There may be multiple instances on an image, such as people, buildings, animals, plants, white clouds, kites, etc. appearing in a certain frame of a certain video. These can all be regarded as instances in the frame. The user can select any instance on the image screen to operate.

After the combined mark is obtained in step S104, it means that the area to be removed in the target frame image has been automatically removed. At this time, the target frame image can be repaired by using existing image restoration methods, such as deep flow-guided video inpainting, flow-edge guided video completion, and temporally coherent completion of dynamic video, etc., which will not be repeated here.

According to people's photography habits, the main body of the photographed object occupies the largest part in the entire video screen (that is, from the overall point of view, the main body of the photographed object appears in the largest number of frames and appears in the video for the longest time), and runs through the entire video screen. Since the method of the above embodiment selects multiple auxiliary frame images in the input video, in steps S101-S102, by comparing the consistency of the reverse optical flow and the forward optical flow in each pair of optical flows, it is possible to approximately obtain the consistency of the target frame image with the screen content in the entire video length, and then compare it with the segmentation marks corresponding to the instances in the target frame image, it is possible to obtain which instances in the current target frame image are non-shooting subjects and which areas are the results that need to be removed. On this basis, the segmentation marks with a degree of overlap higher than the preset condition are retained and combined with the consistency marks. The obtained combined marks are the identification results after removing the areas to be removed.

The embodiment of the present application utilizes the time series relationship in the video, and can automatically identify non-subject objects that need to be removed in the input video and automatically remove areas that need to be removed in the image, thereby achieving automatic repair of the image/video with good repair effect, fast processing speed, manpower saving, and easy use.

Compared with traditional methods, the method of the embodiment of the present application has a faster computing speed and can give calculation results in real time in actual scenarios; compared with other methods using neural networks, the method of the embodiment of the present application can automatically calculate non-subject object labels and give robust and complete labels, thereby greatly reducing the manpower cost required in image editing.

In an exemplary embodiment, in the above step S100, when selecting a target frame image and multiple auxiliary frame images in the input video, all images are selected as auxiliary frame images at fixed intervals in the input video with the target frame image as the center.

Take the target frame image as the first frame image in the input video as an example for explanation. In this case, auxiliary frame images are selected at intervals of s, 2s, ..., ns from the target frame image, where s can represent the number of frames, that is, one auxiliary frame image is selected every s frames from the target frame image, s can also represent the time interval between two adjacent frames, that is, one auxiliary frame image is selected every s time interval from the target frame image, and s can also represent other similar meanings. This embodiment does not limit the meaning of the fixed interval s. The nsth frame image is the last auxiliary frame image that can be selected in the video, that is, all auxiliary frame images that meet the fixed interval in the input video are selected.

On the one hand, in order to ensure that there is a large difference in the picture content between the auxiliary frame image and the target frame image, s cannot be too small; on the other hand, in order to ensure that the number of auxiliary frames is not too small, s cannot be too large. In one embodiment, s is selected so that the frame rate of the auxiliary frame is 6fps. In other embodiments, it can also take any value between 3fps and 12fps, and the embodiment of the present application does not limit this.

When the selected target frame image is an image located in the middle of the input video, the target frame image of the input video can be used as the center, and fixed intervals can be calculated forward and backward to select auxiliary frame images. The selection process can refer to the above example. When the target frame image is selected as the last frame image in the input video, the selection process is similar to the above example and will not be repeated here.

In other embodiments, a fixed number of frames or time intervals may not be used when selecting auxiliary frame images. For example, selection may be performed randomly or with a specified number of frames or in other ways, and the embodiments of the present application do not impose any restrictions on this.

In the above step S101, after pairing the target frame image with multiple auxiliary frame images respectively, each auxiliary frame image corresponds to the target frame image, which is convenient for subsequent comparison to obtain information. The number of pairing combinations is the same as the number of selected auxiliary frame images. In an exemplary embodiment, the consistency mark of the image includes a consistency mark of each grid point position in the target frame image; according to the consistency mark of the grid point position contained in the target frame, the consistency mark of the target frame image can be obtained.

According to the consistency judgment results of the reverse optical flow and the forward optical flow in each pair of optical flows, the consistency mark of the image is obtained, including:

According to the consistency judgment result between the grid point position in the target frame image and the corresponding grid point position in each auxiliary frame image, a consistency mark of the grid point position is obtained, including:

For each pairing combination, an initial consistency mark of the grid point position in the pairing combination is obtained according to the consistency judgment result between the grid point position of the target frame image in the pairing combination and the corresponding grid point position in the auxiliary frame image;

The initial consistent marks of the grid point position in all pairing combinations are accumulated, and the consistent mark of the grid point position is obtained according to the accumulated result.

Assume that the number of auxiliary frame images selected is n. The corresponding number of pairing combinations is also n. After comparing the position of each grid point in the target frame image with the corresponding grid point position of each auxiliary frame image, n initial consistency marks are obtained for each grid point position of the target frame image. Then, these n initial consistency marks of each grid point position are accumulated. After the accumulation, each grid point position corresponds to only one consistency mark.

In an exemplary embodiment, for each pairing combination, obtaining the initial consistency mark of the grid point position in the pairing combination according to the consistency judgment result between the grid point position of the target frame image in the pairing combination and the corresponding grid point position in the auxiliary frame image includes:

According to each pair of optical flows, a new grid point position in the corresponding pairing combination is obtained; the new grid point position is calculated based on the grid point position in the auxiliary frame image in the pairing combination corresponding to the grid point position in the target frame image in combination with the forward optical flow;

For each pairing combination, an initial consistency mark of the grid point position in the pairing combination is obtained according to the consistency judgment result between the grid point position in the target frame image and the new grid point position in the pairing combination.

In this embodiment, by judging the consistency between a certain grid point position of the target frame image and the new grid point position corresponding to the grid point position, the optical flow consistency mark between the forward optical flow and the reverse optical flow can be obtained. Other methods can also be used to judge the consistency between the forward optical flow and the reverse optical flow, such as the neural network fitting method, etc., which is not limited in this embodiment of the present application. The initial consistency mark obtained can be in the form of an image, that is, an optical flow consistency mark map. When the number of auxiliary frame images selected is n, the n initial consistency marks obtained are n optical flow consistency mark maps. The above-mentioned calculation of the consistency mark is a calculation based on the target frame image.

In an exemplary embodiment, after pairing the target frame image and the auxiliary frame image respectively, the target frame image and the auxiliary frame image are used as input, and the optical flow is extracted through the optical flow network, and the optical flow network outputs the optical flow from the target frame image to the auxiliary frame image and from the auxiliary frame image to the target frame image. For each pair of target frame image and auxiliary frame image, a pair of optical flows can be obtained.

In one implementation, the optical flow network may use FlowNet2.0. The use of this network can achieve both high accuracy and high speed, and can complete the extraction of optical flow in a shorter time. In order to obtain the bidirectional optical flow between the target frame image and the auxiliary frame image, the target frame and the auxiliary frame can be input first, and then input in the reverse order to obtain two optical flows output by the network.

The embodiments of the present application do not limit the method of extracting optical flow and the type of optical flow network used.

In an exemplary embodiment, obtaining a new grid point position in a corresponding pairing combination according to each pair of optical flows includes:

For each pair of optical flows, the grid point position of the target frame image is propagated to the auxiliary frame image according to the reverse optical flow, and the new grid point position is calculated according to the grid point position corresponding to the grid point position of the target frame image on the auxiliary frame image and the forward optical flow;

For each pairing combination, respectively according to the consistency judgment result between the grid point position in the target frame image and the new grid point position in the pairing combination, obtaining the initial consistency mark of the grid point position in the pairing combination includes:

For each pairing combination, the following processing is performed:

Compare the grid point position in the target frame image with the difference between the grid point position of the new grid point position in the paired combination after being transmitted back to the target frame image according to the forward optical flow; when the difference is less than or equal to a preset first threshold, record the initial consistency mark of the grid point position as a value indicating consistency.

For each pair of optical flows, the grid point positions of the target frame image are first propagated to the auxiliary frame image according to the reverse optical flow, and the new grid point positions are calculated according to the positions of the grid point positions of the target frame image on the auxiliary frame image and the forward optical flow, and then the new grid point positions are propagated back to the target frame image according to the forward optical flow, and the difference values between the new grid point positions and the grid point positions of the target frame image are compared, and the positions where the difference values are less than or equal to the preset first threshold are marked as consistent. The grid point positions where the difference values are greater than the preset first threshold are considered inconsistent. The case where the difference value is equal to the preset first threshold can be set as needed, and this embodiment does not limit this. Other methods can also be used to compare the consistency of the grid point positions corresponding to the target frame image and the auxiliary frame image, and this embodiment of the present application does not limit this.

In an exemplary embodiment, calculating new grid point positions according to the positions of the grid point positions of the target frame image on the auxiliary frame image and the forward optical flow includes:

The coordinate value of the position of the grid point of the target frame image on the auxiliary frame image is added to the coordinate value of the forward optical flow at the position to obtain the coordinate of the new grid point position.

In an exemplary embodiment, the difference value comprises a Euclidean distance.

After the grid point positions are propagated from the target frame image to the auxiliary frame image, the optical flow of each grid point (i.e., the coordinates of the new grid point positions) can be estimated using the bilinear interpolation method. In other embodiments, the nearest neighbor, cubic interpolation, and other methods can also be used for calculation, which is not limited in the embodiments of the present application. After the new grid point positions are transmitted back to the target frame image, the consistency is determined based on whether the Euclidean distance between the new grid point position and the original grid point position is less than a certain threshold (i.e., the first threshold). The first threshold determines the strictness of the consistency mark and can be set as needed. In this embodiment, the first threshold can be set to 2.

In an exemplary embodiment, obtaining a consistency mark of the grid point position according to the accumulation result includes:

The accumulated result is multiplied by the maximum optical flow mark of the target frame image to obtain a first mark, and the first mark is threshold-cut off to obtain a binary consistency mark of the grid position; wherein the maximum optical flow mark of the target frame image is the maximum value of the modulus of the optical flow of each pixel on the target frame image.

In an exemplary embodiment, the maximum optical flow mark of the target frame image is calculated in the following way:

For each pair of reverse optical flows, the modulus of the optical flow at each pixel is calculated respectively, and the maximum modulus of all optical flows at each pixel is taken as the maximum optical flow mark of the target frame image.

When the number of selected auxiliary frame images is n, there are n reverse optical flow maps, and the modulus length of the optical flow at each pixel of the target frame image also has n values. By comparing these n values, the maximum optical flow mark at each pixel can be obtained. That is, after calculating each pair of optical flows, an optical flow mark map (or optical flow modulus length map) can be obtained. Combining all optical flow marks or optical flow mark maps, a maximum optical flow mark map can be obtained.

In an exemplary embodiment, the modulus of the optical flow is calculated using the following formula:

Where, l _i represents the modulus of the reverse optical flow in the i-th pair of optical flows, x _i and y _i represent the optical flow size (or optical flow speed) of each pixel in the horizontal and vertical directions respectively;

In an exemplary embodiment, the maximum optical flow mark is calculated using the following formula:

L _max =max (L ₁ , L ₂ ,..., L _n ),

Wherein, L represents the optical flow modulus length map, and n represents the total number of optical flow modulus length maps.

In an exemplary embodiment, in the above step S102, obtaining a segmentation mark corresponding to each instance in the target frame image includes:

The target frame image is used as the input of the instance segmentation network, which extracts and outputs the segmentation labels corresponding to all instances in the target frame image.

When the target frame image contains multiple instances, multiple segmentation marks, that is, multiple segmentation mark maps, can be obtained.

In this embodiment, the instance segmentation network uses Mask RCNN. In other embodiments, other instance segmentation networks can also be used. The network takes an image as input and can output segmentation marks corresponding to all instances in the image. For example, when the shooting scene is centered on a person and it is desired to remove non-subject persons, the segmentation mark corresponding to the "person" class can be selected from the segmentation marks.

Using deep networks to perform optical flow prediction and instance segmentation can ensure faster processing speed, making real-time applications of automatic editing possible.

In an exemplary embodiment, performing threshold truncation processing includes:

A second threshold is used to perform threshold truncation processing, where the second threshold is set to be a multiple of the mean of the first marker, and the mean of the first marker is a quotient of the first marker and the number of paired combinations.

The second threshold may also be set in other ways, which are not limited in the present application. By adjusting the second threshold, the result of the binary consistency mark may be adjusted, and the second threshold may be reasonably set as needed to obtain a more accurate result, thereby helping to achieve a better image/video restoration effect.

In an exemplary embodiment, a segmentation mark with a degree of overlap higher than a preset condition is combined with a consistency mark to obtain a combined mark, including:

The degree of coincidence between each segmentation mark and the binary consistency mark is determined respectively, and the segmentation marks with a degree of coincidence higher than a preset condition are added to the binary consistency mark to obtain a combined mark.

In an exemplary embodiment, the degree of overlap is the degree of overlap of the overlapping area of each segmentation mark and the consistency mark, and the preset condition is a third threshold value, which is set to a multiple of the area value of the segmentation mark currently being judged.

The third threshold may also be set in other ways, which is not limited in the embodiment of the present application.

In an exemplary embodiment, the combined mark may be calculated using the following method:

First, all initial consistency marks are recorded as Multiple initial consistency marks are accumulated using the following formula:

Then, multiply the accumulated result by the maximum optical flow marker L _max :

^mfc = ^mcon · _Lmax .

The obtained result is subjected to threshold truncation to obtain a binary optical flow consistency mark.

In this embodiment, a multiple of the pixel mean value of m ^fc is used as the second threshold, and a scaling factor for determining the multiple is set to 2, but this embodiment of the present application is not limited to this.

Finally, for each segmentation mark of the target frame image, the size of the overlapping area between the segmentation mark and the binary optical flow consistency mark is calculated respectively, and compared with the overall size of the segmentation mark. If the size of the overlapping area is larger than a certain multiple of the overall area of the segmentation mark, the segmentation mark is added to the binary optical flow consistency mark, and the scaling factor that determines the multiple is set to 0.3, which is not limited in this embodiment of the present application.

The method of the embodiment of the present application combines consistency tags, optical flow tags, and segmentation tags, and can automatically give tags for non-subject objects in the video, so that image editing can be completed without manual intervention. Due to the combination of multiple tags, the final generated combined tag is more robust and complete. The method of the embodiment of the present application can be applied in the scene of automatic image/video editing of mobile phone photography.

The embodiment of the present application also provides a video image repair method, including: the video image editing method in the above embodiment, the method further includes:

Each auxiliary frame image is propagated according to the forward optical flow, and the required pixels in the propagation result are selected according to the combined mark. The average of all selected pixels at the corresponding positions is taken as the filling content and filled into the target frame image to obtain the repaired image.

This method is performed after the above step S103. For each pair of target frame image and auxiliary frame image, the auxiliary frame image is propagated according to the forward optical flow, and according to the combined mark calculated in step S103, if the pixel of the propagation result is in the combined mark, it is saved, otherwise it is discarded; finally, the average of all selected pixels at the corresponding position is taken as the filling content to fill in the target frame image to complete the repair.

The repair method of the embodiment of the present application can automatically and accurately eliminate non-subject objects in the video and fill in content that is consistent in both space and time. Using multiple markers such as consistency markers, optical flow markers, and segmentation markers, non-subject objects in the video can be stably and completely identified; the repair method based on optical flow propagation can use the previous and next information in the video, so that the repair result has good consistency in space and time, and minimizes the appearance of artificial traces to ensure that the repair result is complete and natural. It can be applied to automatic image/video editing of mobile phone photography.

As shown in Figure 2, an embodiment of the present application also provides a video image editing device, including a memory and a processor, the memory is used to store a program for video image editing; the processor is used to read the program for video image editing and execute the video image editing method in any of the above embodiments.

As shown in Figure 2, an embodiment of the present application also provides a video image repair device, including a memory and a processor, the memory is used to store a program for video image repair; the processor is used to read the program for video image repair and execute the video image repair method in any of the above embodiments.

An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the video image editing or video image repair method in any of the above embodiments.

As shown in FIG3 , the video image editing and repairing method according to an embodiment of the present application is described below with reference to Example 1.

Example 1:

S200 selects a target frame image and multiple auxiliary frame images in the input video;

S201 pairs a target frame image with multiple auxiliary frame images respectively, and extracts a pair of optical flows for each pairing combination;

In this example, the target frame image is selected as the first frame image in the input video; the auxiliary frame images are selected in the order of s, 2s, ..., ns with respect to the target frame image, where s is selected so that the frame rate of the auxiliary frame is 6fps, and ns is the last auxiliary frame image that can be obtained in the input video.

After selecting the image, the target frame image and the auxiliary frame image are paired respectively. The target frame image and the auxiliary frame image are used as input, and the optical flow is extracted through the optical flow network FlowNet2.0. When inputting, first input in the order of target frame and auxiliary frame, and then input in the reverse order to obtain the two optical flows from the target frame image to the auxiliary frame image and from the auxiliary frame image to the target frame image output by the optical flow network.

That is, for each pair of target frame and auxiliary frame, a pair of optical flows can be obtained, namely, reverse optical flow and forward optical flow; the reverse optical flow is the optical flow from the target frame image to the auxiliary frame image in the paired combination, and the forward optical flow is the optical flow from the auxiliary frame image in the paired combination to the target frame image.

S202 respectively obtains the initial consistency mark of each pair of optical flows;

For each pair of optical flows, first propagate the grid point positions of the next frame image to the previous frame image according to the reverse optical flow, use the bilinear interpolation method to estimate the optical flow of each grid point, and get the new grid point position, then propagate the new grid point position back to the next frame image, and determine whether the Euclidean distance between the new grid point position and the grid point position is less than the first threshold. In this example, the first threshold is set to 2. If the Euclidean distance between the new grid point position and the grid point position is greater than the first threshold, then these grid point positions are considered inconsistent, and the other grid point positions are marked as consistent. For each pair of optical flows, an optical flow consistency marking map can be calculated.

Among them, the latter frame image is the auxiliary frame image, and the former frame image is the target frame image, so the calculated consistency mark is based on the target frame image.

S203 calculates the maximum optical flow mark of the target frame image;

Take the reverse optical flow in each pair of optical flows, calculate the modulus of the optical flow at each pixel, and take the value with the longest modulus among all optical flows at each pixel. Combining all the optical flow information, we can get a maximum optical flow label map.

For each pixel on the optical flow of each target frame, there are horizontal and vertical optical flow velocities x _i , y _i , and the optical flow modulus is calculated by the following formula:

Wherein, _li represents the modulus of the reverse optical flow in the i-th pair of optical flows, and _xi and _yi represent the optical flow size (or optical flow speed) of each pixel in the horizontal and vertical directions, respectively.

All optical flow modulus length maps are recorded as L ₁ , L ₂ , ..., L _n , where n represents the total number of optical flow modulus length maps, and the longest modulus length value at each pixel is calculated by the following formula:

L _max =max (L ₁ , L ₂ ,..., L _n ),

Where _Lmax represents the maximum optical flow mark.

S204 obtains the segmentation mark corresponding to each instance in the target frame image;

Taking the target frame image as input, the instance segmentation network Mask RCNN is used to extract the segmentation mark corresponding to each instance, and the instance segmentation network outputs the segmentation mark corresponding to all instances in the target frame. For a target frame image, one or more instance segmentation mark maps may be obtained.

This example does not limit the execution order of the above steps S202-S203, which can be selected as needed; the execution order of step S204 can be executed at any time after step S200 and before step S205, and this example does not limit this.

S205 calculates the combined mark;

First, all initial consistency marks are recorded as Multiple initial consistency marks are accumulated using the following formula:

Then, multiply the accumulated result by the maximum optical flow marker L _max :

^mfc = ^mcon · _Lmax .

The obtained result is subjected to threshold truncation processing to obtain a binary optical flow consistency mark, and the multiple of the pixel mean of ^mfc is used as the second threshold, and the scaling factor that determines the multiple is set to 2.

Finally, for each segmentation mark of the target frame image, the size of the overlapping area between the segmentation mark and the binary optical flow consistency mark is calculated, and compared with the overall size of the segmentation mark. If the size of the overlapping area is greater than 0.3 times the overall area of the segmentation mark, the segmentation mark is added to the binary optical flow consistency mark. After comparing each segmentation mark, a combined mark is finally obtained.

S206 performs optical flow propagation and repairs the target frame image according to the combined mark.

For each pair of target frame image and auxiliary frame image, a corresponding pair of optical flows has been calculated in step S201. The auxiliary frame image is propagated according to the forward optical flow. According to the combined mark calculated in step S205, if the pixel of the propagation result is in the combined mark, it is saved, otherwise it is discarded; finally, the average of all selected pixels at the corresponding positions is taken as the filling content and filled into the target frame image to complete the repair.

It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules/units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims (14)

1. A video image editing method, comprising:

Selecting a frame of target frame image and a plurality of frames of auxiliary frame images in an input video;

Pairing the target frame image and the multi-frame auxiliary frame image respectively to obtain a plurality of pairing combinations, wherein each pairing combination comprises the target frame image and one frame of auxiliary frame image;

Extracting a pair of optical flows for each pairing combination respectively, including: reverse optical flow and forward optical flow; the reverse optical flow is the optical flow from the target frame image to the auxiliary frame image in the pairing combination, and the forward optical flow is the optical flow from the auxiliary frame image to the target frame image in the pairing combination;

Obtaining consistency marks of the images according to consistency judgment results of the reverse optical flow and the forward optical flow in each pair of optical flows; the consistency marks of the images comprise consistency marks of the positions of each grid point in the target frame image;

obtaining a segmentation mark corresponding to each instance in the target frame image;

Determining the coincidence degree of the segmentation marks corresponding to each instance and the consistency marks respectively, and combining the segmentation marks with the coincidence degree higher than a preset condition with the consistency marks to obtain a combined mark, wherein the combined mark is used for marking the result of removing the region to be removed from the target frame image;

the step of obtaining the consistency mark of the image according to the consistency judgment result of the reverse optical flow and the forward optical flow in each pair of optical flows comprises the following steps:

For each pairing combination, respectively obtaining an initial consistency mark of the grid point position in the pairing combination according to a consistency judgment result between the grid point position of the target frame image in the pairing combination and the corresponding grid point position in the auxiliary frame image;

And accumulating the initial consistency marks of the grid point positions in all the pairing combinations, and obtaining the consistency marks of the grid point positions according to the accumulation result.

2. The video image editing method according to claim 1, wherein for each pairing combination, obtaining an initial consistency flag of the grid point position in the pairing combination according to a consistency determination result between the grid point position of the target frame image in the pairing combination and the corresponding grid point position in the auxiliary frame image, respectively, includes:

Obtaining new lattice point positions in corresponding pairing combinations according to each pair of light flows respectively; the new lattice point position is calculated according to the combination of the lattice point position corresponding to the lattice point position of the target frame image in the auxiliary frame image in the pairing combination and the forward optical flow;

And for each pairing combination, acquiring an initial consistency mark of the grid point position in the pairing combination according to a consistency judgment result between the grid point position in the target frame image and a new grid point position in the pairing combination.

3. The video image editing method according to claim 2, wherein obtaining new lattice positions in the corresponding pairing combinations from each pair of light streams, respectively, comprises:

For each pair of optical flows, propagating the grid point position of the target frame image to the auxiliary frame image according to the reverse optical flow, and calculating the new grid point position according to the corresponding grid point position of the target frame image on the auxiliary frame image and the forward optical flow;

For each pairing combination, respectively obtaining the initial consistency mark of the grid point position in the pairing combination according to the consistency judgment result between the grid point position in the target frame image and the new grid point position in the pairing combination comprises the following steps:

the following treatments were performed for each pairing combination, respectively:

And comparing the grid point position in the target frame image, and returning the new grid point position in the pairing combination to a difference value between the grid point positions on the target frame image according to the forward optical flow, and recording an initial consistency mark of the grid point position as a value representing consistency when the difference value is smaller than or equal to a preset first threshold value.

4.A video image editing method according to claim 3, wherein the difference value comprises a euclidean distance.

5. The video image editing method according to claim 2, wherein said obtaining the consistency flag of the lattice point position according to the accumulation result comprises:

multiplying the accumulated result with the maximum optical flow mark of the target frame image to obtain a first mark, and carrying out threshold value truncation processing on the first mark to obtain the binarized consistency mark of the grid point position; wherein the maximum optical flow of the target frame image is marked as the maximum value of the modular length of the optical flow of each pixel point on the target frame image.

6. The video image editing method according to claim 5, wherein the maximum optical flow label of the target frame image is calculated by:

And respectively calculating the modulo length of the optical flow on each pixel point for the backward optical flow in each pair of optical flows, and taking the maximum value of the modulo length of all the optical flows on each pixel point as the maximum optical flow mark of the target frame image.

7. The video image editing method according to claim 5, wherein said performing a threshold truncation process includes:

And carrying out threshold truncation processing by adopting a second threshold, wherein the second threshold is set to be a multiple of the average value of the first mark, and the average value of the first mark is the quotient of the number of the first mark and the pairing combination.

8. The video image editing method according to claim 5, wherein said combining the split mark with the overlap ratio higher than a preset condition with the consistency mark to obtain a combined mark comprises:

And respectively judging the coincidence degree of each segmentation mark and the binarized consistency mark, and adding the segmentation mark with the coincidence degree higher than the preset condition into the binarized consistency mark to obtain the combined mark.

9. The video image editing method according to claim 8, wherein the overlap ratio is an overlap ratio of an overlapping region of each of the division marks and the consistency mark, the preset condition is a third threshold value set as a multiple of an area value of the division mark currently being judged.

10. The video image editing method according to claim 1, wherein selecting a target frame image and a multi-frame auxiliary frame image in the input video comprises:

And taking the target frame image as a center, and selecting all images in the input video at fixed intervals as auxiliary frame images.

11. A method for video image restoration, comprising: the video image editing method of any of the preceding claims 1-10, the method further comprising:

and respectively spreading the auxiliary frame images of each frame according to the forward optical flow, selecting the required pixels in the spreading result according to the combined mark, taking the average value of all the selected pixels at the corresponding positions as filling content, and filling the filling content into the target frame image to obtain the repaired image.

12. A video image editing apparatus comprising a memory and a processor, wherein the memory is for storing a program for performing video image editing; the processor is configured to read the program for video image editing and execute the video image editing method according to any one of claims 1 to 10.

13. A video image restoration device comprising a memory and a processor, wherein the memory is used for storing a program for performing video image restoration; the processor is configured to read the program for performing video image restoration and perform the video image restoration method according to claim 11.

14. A computer readable storage medium storing computer executable instructions for performing the method of any one of claims 1-10 or claim 11.