CN115118948A - Method and device for repairing irregular occlusion in panoramic video - Google Patents
Method and device for repairing irregular occlusion in panoramic video Download PDFInfo
- Publication number
- CN115118948A CN115118948A CN202210695317.6A CN202210695317A CN115118948A CN 115118948 A CN115118948 A CN 115118948A CN 202210695317 A CN202210695317 A CN 202210695317A CN 115118948 A CN115118948 A CN 115118948A
- Authority
- CN
- China
- Prior art keywords
- video
- frame
- video frame
- occlusion
- repairing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The application discloses repair method and device of random sheltering from in panoramic video, through the video frame of the random region of sheltering from of existence that selects out panoramic video, thereby realize the screening that there is the video frame of random region of sheltering from in the panoramic video, in order to realize accurate restoration, reduce prosthetic quantity, and last video frame and the next video frame according to the video frame, restore the region of sheltering from in the video frame, thereby can break away from artificial operation, realize automatic restoration panoramic video, and the last frame and the next frame that utilize the video frame that needs the restoration carry out the restoration of current frame, utilize adjacent frame to restore the current frame promptly, in order to guarantee the authenticity and the accuracy of the current frame that the restoration obtained.
Description
Technical Field
The application relates to the technical field of video image restoration, in particular to a method and a device for restoring irregular occlusion in a panoramic video.
Background
With the continuous development of VR technology, VR panoramic video is also more and more adopted by people. Outdoor scenes usually use helicopter-mounted dome camera equipment to shoot continuous long-lens materials, however, due to the fact that VR panoramic videos may cause sporadic and irregular shelters to easily appear in the aerial shooting mode due to the view angle or other reasons in the shooting process. The irregularity means that the attribution type of the obstruction is unknown and the appearance position is unknown. The appearance of the shelter causes bad film appearance and strong feeling of being obtrusive. In order to solve the problem, the image can be restored, in an actual implementation of a post-restoration scheme, the panoramic video has nonlinear distortion, while the conventional restoration method is basically performed under the condition of no distortion or little distortion, and the special characteristic causes that the panoramic video has higher restoration difficulty and higher requirements on restoration participants and restoration methods compared with the undistorted video.
Disclosure of Invention
The present application is proposed to solve the above-mentioned technical problems. The embodiment of the application provides a method and a device for repairing irregular occlusion in a panoramic video, and solves the technical problem.
According to an aspect of the present application, a method for repairing irregular occlusion in a panoramic video is provided, including: screening out video frames with irregular shielding areas in the panoramic video; and repairing the occlusion area in the video frame according to the previous video frame and the next video frame of the video frame.
In an embodiment, the screening out the video frames in the panoramic video where the irregular occlusion areas exist includes: decoding the panoramic video to obtain a plurality of frames of images; calculating to obtain a forward light flow graph of the multi-frame image by taking a front frame as a base; calculating based on the post frame to obtain a reverse light flow diagram of the multi-frame image; and screening out video frames with irregular shielding areas in the multi-frame images according to the forward light flow graph and the reverse light flow graph.
In an embodiment, the screening, according to the forward light flow graph and the reverse light flow graph, video frames in which an irregular occlusion area exists in the multi-frame image includes: and determining a video frame with an intersection of the void areas in the forward optical flow graph and the reverse optical flow graph as the video frame with the irregular occlusion area.
In an embodiment, the screening out the video frames in the panoramic video where the irregular occlusion area exists includes: decoding the panoramic video to obtain a plurality of frames of images; respectively calculating the similarity between each frame of image in the multiple frames of images and the corresponding previous frame of image and the next frame of image; and determining the image with the similarity smaller than a preset similarity threshold value as the video frame with the irregular occlusion area.
In an embodiment, after the screening out the video frames in the panoramic video where the irregular occlusion areas exist, the method for repairing the irregular occlusion in the panoramic video further includes: converting the video frame into a grayscale image; carrying out binarization processing on the gray level image; performing edge detection on the gray level image after binarization processing to obtain a region to be repaired; wherein, according to a previous video frame and a next video frame of the video frames, repairing an occlusion region in the video frame comprises: and repairing the area to be repaired according to the previous video frame and the next video frame of the video frames.
In an embodiment, the performing edge detection on the binarized grayscale image to obtain an area to be repaired includes: calculating to obtain an external square of the gray level image after binarization processing; expanding outwards by taking the center of the external square as a circular point and the diagonal length of the external square as an initial diameter to obtain a communicated area with consistent gray value; and setting the gray values in the communication area and outside the communication area as different values respectively to obtain the area to be repaired.
In an embodiment, the expanding outward with the center of the circumscribed square as a circular point and the diagonal length of the circumscribed square as an initial diameter to obtain a connected region with a consistent gray value includes: and expanding outwards by taking the center of the external square as a circular point, taking the diagonal length of the external square as an initial diameter and a preset expansion step length to obtain a maximum communication area with consistent gray values.
In an embodiment, the repairing the occlusion region in the video frame according to the previous video frame and the next video frame of the video frames comprises: and inputting the video frame, the previous video frame of the video frame and the next video frame of the video frame into a neural network model to obtain target area data.
In an embodiment, the repairing an occlusion region in the video frame according to a previous video frame and a next video frame of the video frames further includes: calculating a central point corresponding to the target area data; and fusing the target area data into the video frame from the central point according to a gradient consistency principle to obtain the repaired video frame.
According to another aspect of the present application, there is provided a device for repairing irregular occlusion in a panoramic video, including: the screening module is used for screening out video frames with irregular shielding areas in the panoramic video; and the repairing module is used for repairing the sheltered area in the video frame according to the last video frame and the next video frame of the video frame.
The application provides a repair method and device for irregular occlusion in panoramic video, through screening out the video frame that has the irregular occlusion area in the panoramic video, thereby realize the screening that has the video frame in the irregular occlusion area in the panoramic video, in order to realize accurate restoration, reduce prosthetic quantity, and according to the last video frame and the next video frame of video frame, restore the occlusion area in the video frame, thereby can break away from artificial operation, realize the automatic restoration panoramic video, and utilize the last frame and the next frame that need prosthetic video frame to carry out the restoration of current frame, utilize adjacent frame to restore the current frame promptly, in order to guarantee authenticity and the accuracy of the current frame that the restoration obtained.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally represent like parts or steps.
Fig. 1 is a schematic flowchart of a method for repairing irregular occlusion in a panoramic video according to an exemplary embodiment of the present application.
Fig. 2 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application.
Fig. 3 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application.
Fig. 4 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application.
Fig. 5 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application.
Fig. 6 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application.
Fig. 7 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application.
Fig. 8 is a schematic structural diagram of a repairing apparatus for irregular occlusion in a panoramic video according to an exemplary embodiment of the present application.
Fig. 9 is a schematic structural diagram of a repairing apparatus for irregular occlusion in a panoramic video according to another exemplary embodiment of the present application.
Fig. 10 is a block diagram of an electronic device provided in an exemplary embodiment of the present application.
Detailed Description
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and that the present application is not limited by the example embodiments described herein.
There may be three methods for video occlusion detection:
one is a feature-based object detection method, which has the significant disadvantage that the class of the detected object must be known.
The second is an observation-based content perception method, which is the most straightforward way to identify an unknown occurrence of an occlusion by the human eye, but this means that a great deal of human intervention is required.
Thirdly, the mixture of the two methods, namely, the determination of the class of the obstruction by observation and the determination of the position of the obstruction by feature method, but in most cases, the class of the obstruction is not common or the obstruction class is unavailable due to motion blur, which means that the manual feature extraction method or the feature extraction method based on learning has problems.
Occlusion removal of images generally has three ideas: firstly, through spatial domain information, similar patch in an image is searched for filling a sheltered area based on content identification or approximate matching, the method is suitable for sheltering regular objects or weak textures, but the selection of the size of the patch and the similar area needs manual intervention, and meanwhile, the sheltered area sometimes has spatial domain uniqueness; secondly, through time domain information, the thinking is similar to a space domain method, and the difference is that the content used for filling is searched based on context feature matching, so that the problem of small space domain search range is solved to a certain extent, but manual intervention is still needed; and thirdly, texture information is created through the adjacent area of the shielded area, and the method is suitable for the situation that the shielded area is small, and the distortion phenomenon is easy to occur after the shielded area is large.
In order to overcome the technical difficulties, the application provides a method and a device for repairing irregular occlusion in a panoramic video, images with irregular occlusion areas are screened by using an algorithm, and adjacent frame images are adopted for repairing, so that manual intervention is not needed, and more accurate images can be obtained through repairing. Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Fig. 1 is a schematic flowchart of a method for repairing irregular occlusion in a panoramic video according to an exemplary embodiment of the present application. As shown in fig. 1, the method for repairing irregular occlusion in a panoramic video includes the following steps:
step 110: and screening out video frames with irregular shielding areas in the panoramic video.
The irregular occlusion means that occlusion regions appear on an image frame sequence or appear continuously, but the appearance positions do not have an overlapping phenomenon in adjacent frames, or the occlusion regions appear in the image frame sequence at intervals, if an occlusion exists in a certain frame, no occlusion exists in the front and rear frames. And screening out the video frames with irregular shielding areas in the panoramic video through the irregular shielding characteristics so as to obtain the video frames needing to be repaired.
Step 120: and repairing the occlusion area in the video frame according to the previous video frame and the next video frame of the video frame.
After a video frame containing a random occlusion area is detected, the video frame and related frames (namely a previous video frame and a next video frame) of the video frame are selected as input, data lacking in the video frame are extracted in the input to generate a composite frame based on an optical flow data extraction method, and then the extracted and generated data corresponding to the random occlusion area are fused into the video frame to complete the repair of the random occlusion area of the video frame.
The application provides a repair method of random sheltering from in panoramic video, through the video frame of the random region of sheltering from of existence in screening out the panoramic video, thereby realize the screening that there is the video frame of random region of sheltering from in the panoramic video, in order to realize accurate restoration, reduce prosthetic quantity, and last video frame and the next video frame according to the video frame, repair the region of sheltering from in the video frame, thereby can break away from artificial operation, realize automatic restoration panoramic video, and the last frame and the next frame that utilize the video frame that needs the restoration carry out the restoration of current frame, utilize adjacent frame to restore the current frame promptly, in order to guarantee the authenticity and the accuracy of the current frame that the restoration obtained.
Fig. 2 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application. As shown in fig. 2, the step 110 may include:
step 111: and decoding the panoramic video to obtain a plurality of frames of images.
Starting up the detection of irregular sheltering area, firstly decoding the selected video into image frame sequence, and grouping the image frame sequence into a set V { V } through scene detection 1 ,…,V x ,…,V max And Vx represents the xth group, wherein no group contains multiple frame images.
Step 112: and calculating based on the preposed frame to obtain a forward light flow graph of the multi-frame image.
Step 113: and calculating based on the post frame to obtain a reverse light flow diagram of the multi-frame image.
Operating on each group in V in turn, with V x {I 1 ,…,I i ,…I max Take the example that follow is all equal to V x Similarly. First from V x In (1) is selected i Frame and I i+1 Frame (I starts from 1), calculate I i Frame to I i+1 Optical flow of frame F i->i+1 And based on the preposed frame, obtaining a forward optical flow uv graph G i->i+1 . While calculating I i+1 Frame to I i Optical flow of frame F i+1->i And obtaining a backward light flow uv graph G based on the post-positioned frame i+1->i 。
Step 114: and screening out video frames with irregular shielding areas in the multi-frame images according to the forward light flow graph and the reverse light flow graph.
In an embodiment, the specific implementation manner of step 114 may be: and determining a video frame with intersection in the cavity area in the forward optical flow graph and the reverse optical flow graph as a video frame with a random occlusion area. By comparing the optical flow uv maps G i->i+1 And G i+1->i The hole distribution of (2), calculate G i->i+1 And G i+1->i Obtaining a preliminarily screened suspected frame containing an occlusion areaAnd (5) collecting S.
Fig. 3 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application. As shown in fig. 3, the step 110 may include:
step 115: and decoding the panoramic video to obtain a plurality of frames of images.
Step 115 is similar to step 111 described above and will not be described herein. Also, step 115 may be the same step as step 111.
Step 116: and respectively calculating the similarity between each frame of image in the multi-frame of images and the corresponding previous frame of image and the corresponding next frame of image.
Each frame of image may be the multiple frames of images, or may be an image obtained after the screening in steps 111 to 114. Selecting a frame image I from the frame set S in turn i And a corresponding frame sequence V x Frame I in i-1 And I i+1 Then to frame I i Expanding the contained cavity area by 3 to 4 times of the original size to obtain a detection area P i Will detect the region P i Applied to frame I i-1 And I i+1 The same region is obtained as the detection region P i-1 And P i+1 Then, P is calculated respectively i And P i-1 SSIM value and P of i And P i+1 SSIM value of.
Step 117: and determining the image with the similarity smaller than a preset similarity threshold as a video frame with an irregular occlusion area.
When the similarity is smaller than a preset similarity threshold value, it indicates that there is an irregular occlusion area in the image, for example, if P i And P i-1 SSIM value and P of i And P i+1 If the SSIM values are all larger than 0.9, the frame is removed from the frame set S to obtain a final frame set S containing the occlusion region final 。
Fig. 4 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application. As shown in fig. 4, after step 110, the method for repairing a random occlusion in a panoramic video may further include:
step 130: the video frame is converted into a grayscale image.
From the frame set S in turn final Selecting a frame image I i Then the image I is processed i And converted into a gray-scale image.
Step 140: and carrying out binarization processing on the gray level image.
The grayscale map is binarized by setting a threshold (e.g., 20, etc.) to obtain a binarized image.
Step 150: and carrying out edge detection on the gray level image after the binarization processing to obtain an area to be repaired.
And then, using a Canny operator to carry out edge detection, and smoothing the pixel edge through morphological closing operation to obtain the region to be repaired.
Correspondingly, step 120 is adjusted to: and repairing the area to be repaired according to the previous video frame and the next video frame of the video frames.
Fig. 5 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application. As shown in fig. 5, the step 150 may include:
step 151: and calculating to obtain an external square of the gray level image after the binarization processing.
Since the occlusion hole region is a subset of the true occlusion region, the hole region needs to be expanded to reach the true occlusion region. First calculating an image I i And a non-deflected circumscribed square of the hollow region.
Step 152: the center of the external square is a circular point, and the diagonal length of the external square is an initial diameter and expands outwards to obtain a communicated area with consistent gray values.
In an embodiment, the specific implementation manner of step 152 may be: and the center of the external square is a circular point, the length of the diagonal line of the external square is an initial diameter, and the external square is expanded outwards by a preset expansion step length to obtain a maximum communication area with consistent gray values. Specifically, the center of the circumscribed square is taken as a circular point, the diagonal length of the square is taken as an initial diameter, and a region is searched at the growth speed of round (max (image width, image height) × 0.004) pixelsThe largest connected domain in the set, if the connected domain contains I i In the hollow area, the connected domain is selected as the contour edge O of the final irregular shelter i 。
Step 153: and respectively setting the gray values in the communication area and outside the communication area as different values to obtain the area to be repaired.
Is prepared from O i After round (max (image width, image height) × 0.004) pixels are expanded, the area inside is set to 1, and the area outside is set to 0, so that a mask image for circling the irregular obstruction is generated.
Fig. 6 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application. As shown in fig. 6, the step 120 may include:
step 121: and inputting the video frame, the previous video frame of the video frame and the next video frame of the video frame into the neural network model to obtain target area data.
From the frame set S in turn final In which a frame of image I is selected i And corresponding Mask i And from the frame sequence V x Selecting corresponding front and back frames I i-1 And I i+1 Generating target area data R using a trained neural network model for generating target repair area data i g 。
Fig. 7 is a flowchart illustrating a method for repairing a random occlusion in a panoramic video according to another exemplary embodiment of the present application. As shown in fig. 7, the step 120 may further include:
step 122: and calculating the central point corresponding to the target area data.
Step 123: and fusing the target area data into the video frame from the central point according to a gradient consistency principle to obtain a repaired video frame.
First, calculate Mask i Will then generate data R by a gradient consistency method i g Integration into original frame I i In the random occlusion region.
Fig. 8 is a schematic structural diagram of a repairing apparatus for irregular occlusion in a panoramic video according to an exemplary embodiment of the present application. As shown in fig. 8, the apparatus 80 for repairing irregular occlusion in a panoramic video includes: the screening module 81 is used for screening out video frames with irregular shielding areas in the panoramic video; and a repairing module 82, configured to repair the occlusion region in the video frame according to the previous video frame and the next video frame of the video frame.
The application provides a pair of prosthetic devices that random sheltered from in panoramic video, the video frame in the random region of sheltering from of existence in the panoramic video is selected through screening module 81, thereby realize the screening of the video frame in the random region of sheltering from of existence in the panoramic video, in order to realize accurate restoration, reduce prosthetic quantity, and repair module 82 is according to the last video frame and the next video frame of video frame, the region of sheltering from in the restoration video frame, thereby can break away from artificial operation, realize automatic restoration panoramic video, and the last frame and the next frame that utilize the video frame that needs the restoration carry out the restoration of current frame, utilize adjacent frame to restore the current frame promptly, in order to guarantee authenticity and the accuracy of the current frame that the restoration obtained.
Fig. 9 is a schematic structural diagram of a repairing apparatus for irregular occlusion in a panoramic video according to another exemplary embodiment of the present application. As shown in fig. 9, the screening module 81 may include: a decoding unit 811 for decoding the panoramic video to obtain a plurality of frame images; a forward direction calculation unit 812, configured to calculate a forward direction light flow map of the multi-frame image based on the previous frame; a reverse calculation unit 813 configured to calculate a reverse light flow map of the multi-frame image based on the post frame; and the prescreening unit 814 is used for screening out video frames with irregular shielding areas in the multi-frame images according to the forward light flow graph and the reverse light flow graph.
In an embodiment, the intersection calculation unit 814 may be further configured to: and determining a video frame with intersection in the cavity area in the forward optical flow graph and the reverse optical flow graph as a video frame with a random occlusion area.
In an embodiment, as shown in fig. 9, the screening module 81 may further include: a similarity calculation unit 815 for calculating the similarity between each frame of image in the multiple frames of images and the corresponding previous frame of image and the next frame of image; and the rescreening unit 816 is configured to determine that the image with the similarity smaller than the preset similarity threshold is a video frame with a random occlusion area.
In an embodiment, as shown in fig. 9, the apparatus 80 for repairing a random occlusion in a panoramic video may further include: a grayscale processing module 83, configured to convert the video frame into a grayscale image; a binary processing module 84, configured to perform binarization processing on the grayscale image; and the edge detection module 85 is configured to perform edge detection on the binarized grayscale image to obtain an area to be repaired. Correspondingly, the repair module 82 is configured to: and repairing the area to be repaired according to the previous video frame and the next video frame of the video frames.
In one embodiment, as shown in fig. 9, the edge detection module 85 may include: a square calculation unit 851 for calculating an external square of the binarized gray-scale image; the expansion unit 852 is used for outwards expanding by taking the center of the external square as a circular point and the diagonal length of the external square as an initial diameter to obtain a communicated area with a consistent gray value; and a gray level setting unit 853, configured to set gray levels inside and outside the connected region to different values, respectively, so as to obtain the region to be repaired.
In an embodiment, the expansion unit 852 may be configured to: and the center of the external square is a circular point, the length of the diagonal line of the external square is an initial diameter, and the external square is expanded outwards by a preset expansion step length to obtain a maximum communication area with consistent gray values.
In one embodiment, as shown in fig. 9, the repair module 82 may include: and a target data calculating unit 821, configured to input the video frame, a previous video frame of the video frame, and a next video frame of the video frame into the neural network model to obtain target area data.
In one embodiment, as shown in fig. 9, the repair module 82 may further include: a central point calculating unit 822 for calculating a central point corresponding to the target area data; and a data merging unit 823, configured to merge the target area data into the video frame from the central point according to a gradient consistency principle to obtain a repaired video frame.
Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 10. The electronic device may be either or both of the first device and the second device, or a stand-alone device separate from them, which stand-alone device may communicate with the first device and the second device to receive the acquired input signals therefrom.
FIG. 10 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.
As shown in fig. 10, the electronic device 10 includes one or more processors 11 and memory 12.
The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer readable storage medium, and the processor 11 may execute the program instructions to implement the method for repairing irregular occlusion in panoramic video of the various embodiments of the present application described above and/or other desired functions. Various contents such as an input signal, a signal component, a noise component, etc. may also be stored in the computer-readable storage medium.
In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
When the electronic device is a stand-alone device, the input means 13 may be a communication network connector for receiving the acquired input signals from the first device and the second device.
The input device 13 may also include, for example, a keyboard, a mouse, and the like.
The output device 14 may output various information including the determined distance information, direction information, and the like to the outside. The output devices 14 may include, for example, a display, speakers, printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 10 relevant to the present application are shown in fig. 10, and components such as buses, input/output interfaces, and the like are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the method for repairing irregular occlusions in panoramic video according to various embodiments of the present application described in the above-mentioned "exemplary methods" section of this specification.
The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the method for repairing irregular occlusions in panoramic video according to various embodiments of the present application, described in the "exemplary methods" section above in this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, each component or step can be decomposed and/or re-combined. These decompositions and/or recombinations are to be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (10)
1. A method for repairing irregular occlusion in a panoramic video is characterized by comprising the following steps:
screening out video frames with irregular shielding areas in the panoramic video; and
and repairing the occlusion area in the video frame according to the previous video frame and the next video frame of the video frame.
2. The method for repairing irregular occlusion in a panoramic video according to claim 1, wherein the screening out the video frames in the panoramic video having irregular occlusion areas comprises:
decoding the panoramic video to obtain a plurality of frames of images;
calculating to obtain a forward light flow graph of the multi-frame image by taking a front frame as a base;
calculating based on the post frame to obtain a reverse light flow diagram of the multi-frame image; and
and screening out video frames with irregular shielding areas in the multi-frame images according to the forward light flow graph and the reverse light flow graph.
3. The method for repairing irregular occlusion in a panoramic video according to claim 2, wherein the step of screening out the video frames with irregular occlusion areas in the multi-frame images according to the forward light flow graph and the reverse light flow graph comprises:
and determining a video frame with an intersection of the cavity areas in the forward optical flow graph and the reverse optical flow graph as the video frame with the irregular occlusion area.
4. The method for repairing irregular occlusion in a panoramic video according to claim 1, wherein the screening out the video frames in the panoramic video having irregular occlusion areas comprises:
decoding the panoramic video to obtain a plurality of frames of images;
respectively calculating the similarity between each frame of image in the multiple frames of images and the corresponding previous frame of image and the next frame of image; and
and determining the image with the similarity smaller than a preset similarity threshold value as the video frame with the irregular occlusion area.
5. The method for repairing irregular occlusion in a panoramic video according to claim 1, wherein after the screening out the video frames in the panoramic video where the irregular occlusion areas exist, the method for repairing irregular occlusion in a panoramic video further comprises:
converting the video frame into a grayscale image;
carrying out binarization processing on the gray level image; and
performing edge detection on the gray level image after binarization processing to obtain an area to be repaired;
wherein, according to a previous video frame and a next video frame of the video frames, repairing an occlusion region in the video frame comprises:
and repairing the area to be repaired according to the previous video frame and the next video frame of the video frames.
6. The method for repairing irregular occlusion in a panoramic video according to claim 5, wherein the edge detection of the binarized gray-scale image to obtain the region to be repaired comprises:
calculating to obtain an external square of the gray level image after binarization processing;
expanding outwards by taking the center of the external square as a circular point and the diagonal length of the external square as an initial diameter to obtain a communicated area with consistent gray value; and
and setting the gray values in the communication area and outside the communication area as different values respectively to obtain the area to be repaired.
7. The method for repairing irregular occlusion in a panoramic video according to claim 6, wherein the step of expanding outward by taking the center of the circumscribed square as a circular point and the diagonal length of the circumscribed square as an initial diameter to obtain a connected region with a consistent gray value comprises:
and expanding outwards by taking the center of the external square as a circular point, the diagonal length of the external square as an initial diameter and a preset expansion step length to obtain a maximum communication area with consistent gray values.
8. The method for repairing irregular occlusion in a panoramic video according to claim 1, wherein the repairing occlusion areas in the video frames according to a previous video frame and a next video frame of the video frames comprises:
and inputting the video frame, the previous video frame of the video frame and the next video frame of the video frame into a neural network model to obtain target area data.
9. The method for repairing irregular occlusion in panoramic video of claim 8, wherein the repairing occlusion regions in the video frames according to a previous video frame and a next video frame of the video frames further comprises:
calculating a central point corresponding to the target area data; and
and fusing the target area data into the video frame from the central point according to a gradient consistency principle to obtain the repaired video frame.
10. The utility model provides a prosthetic devices of random sheltering from in panoramic video which characterized in that includes:
the screening module is used for screening out video frames with irregular shielding areas in the panoramic video; and
and the repairing module is used for repairing the sheltered area in the video frame according to the last video frame and the next video frame of the video frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210695317.6A CN115118948B (en) | 2022-06-20 | 2022-06-20 | Repairing method and device for irregular shielding in panoramic video |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210695317.6A CN115118948B (en) | 2022-06-20 | 2022-06-20 | Repairing method and device for irregular shielding in panoramic video |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115118948A true CN115118948A (en) | 2022-09-27 |
| CN115118948B CN115118948B (en) | 2024-04-05 |
Family
ID=83327840
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210695317.6A Active CN115118948B (en) | 2022-06-20 | 2022-06-20 | Repairing method and device for irregular shielding in panoramic video |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115118948B (en) |
Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1758095A (en) * | 2004-02-02 | 2006-04-12 | 夏普株式会社 | Liquid crystal display device |
| US20080247649A1 (en) * | 2005-07-07 | 2008-10-09 | Chun Hing Cheng | Methods For Silhouette Extraction |
| US20090129700A1 (en) * | 2005-07-28 | 2009-05-21 | Carsten Rother | Image Blending |
| CN102349294A (en) * | 2009-03-13 | 2012-02-08 | 株式会社理光 | Video editing device and video editing system |
| CN104240235A (en) * | 2014-08-26 | 2014-12-24 | 北京君正集成电路股份有限公司 | Method and system for detecting whether camera is covered or not |
| CN106504282A (en) * | 2016-11-23 | 2017-03-15 | 浙江大华技术股份有限公司 | A kind of video shelter detection method and device |
| CN109829449A (en) * | 2019-03-08 | 2019-05-31 | 北京工业大学 | A kind of RGB-D indoor scene mask method based on super-pixel space-time context |
| CN109903321A (en) * | 2018-10-16 | 2019-06-18 | 迈格威科技有限公司 | Image processing method, image processing apparatus and storage medium |
| CN110728697A (en) * | 2019-09-30 | 2020-01-24 | 华中光电技术研究所（中国船舶重工集团有限公司第七一七研究所） | Infrared dim target detection tracking method based on convolutional neural network |
| CN110782415A (en) * | 2019-11-01 | 2020-02-11 | 合肥图鸭信息科技有限公司 | Image completion method and device and terminal equipment |
| CN111462191A (en) * | 2020-04-23 | 2020-07-28 | 武汉大学 | Non-local filter unsupervised optical flow estimation method based on deep learning |
| CN112862671A (en) * | 2021-02-09 | 2021-05-28 | 清华大学 | Video image editing and repairing method, device and storage medium |
| US20210217145A1 (en) * | 2020-01-14 | 2021-07-15 | Samsung Electronics Co., Ltd. | System and method for multi-frame contextual attention for multi-frame image and video processing using deep neural networks |
| CN113129312A (en) * | 2018-10-15 | 2021-07-16 | 华为技术有限公司 | Image processing method, device and equipment |
| CN113538275A (en) * | 2021-07-14 | 2021-10-22 | 华中科技大学 | A method and system for fruit occlusion recovery based on CycleGAN |
| CN113658231A (en) * | 2021-07-07 | 2021-11-16 | 北京旷视科技有限公司 | Optical flow prediction method, device, electronic device and storage medium |
| US20210407051A1 (en) * | 2020-06-26 | 2021-12-30 | Nvidia Corporation | Image generation using one or more neural networks |
| CN114072840A (en) * | 2020-05-26 | 2022-02-18 | 百度时代网络技术（北京）有限公司 | Depth-guided video repair for autonomous driving |
-
2022
- 2022-06-20 CN CN202210695317.6A patent/CN115118948B/en active Active
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1758095A (en) * | 2004-02-02 | 2006-04-12 | 夏普株式会社 | Liquid crystal display device |
| US20080247649A1 (en) * | 2005-07-07 | 2008-10-09 | Chun Hing Cheng | Methods For Silhouette Extraction |
| US20090129700A1 (en) * | 2005-07-28 | 2009-05-21 | Carsten Rother | Image Blending |
| CN102349294A (en) * | 2009-03-13 | 2012-02-08 | 株式会社理光 | Video editing device and video editing system |
| CN104240235A (en) * | 2014-08-26 | 2014-12-24 | 北京君正集成电路股份有限公司 | Method and system for detecting whether camera is covered or not |
| CN106504282A (en) * | 2016-11-23 | 2017-03-15 | 浙江大华技术股份有限公司 | A kind of video shelter detection method and device |
| CN113129312A (en) * | 2018-10-15 | 2021-07-16 | 华为技术有限公司 | Image processing method, device and equipment |
| CN109903321A (en) * | 2018-10-16 | 2019-06-18 | 迈格威科技有限公司 | Image processing method, image processing apparatus and storage medium |
| CN109829449A (en) * | 2019-03-08 | 2019-05-31 | 北京工业大学 | A kind of RGB-D indoor scene mask method based on super-pixel space-time context |
| CN110728697A (en) * | 2019-09-30 | 2020-01-24 | 华中光电技术研究所（中国船舶重工集团有限公司第七一七研究所） | Infrared dim target detection tracking method based on convolutional neural network |
| CN110782415A (en) * | 2019-11-01 | 2020-02-11 | 合肥图鸭信息科技有限公司 | Image completion method and device and terminal equipment |
| US20210217145A1 (en) * | 2020-01-14 | 2021-07-15 | Samsung Electronics Co., Ltd. | System and method for multi-frame contextual attention for multi-frame image and video processing using deep neural networks |
| CN111462191A (en) * | 2020-04-23 | 2020-07-28 | 武汉大学 | Non-local filter unsupervised optical flow estimation method based on deep learning |
| CN114072840A (en) * | 2020-05-26 | 2022-02-18 | 百度时代网络技术（北京）有限公司 | Depth-guided video repair for autonomous driving |
| US20210407051A1 (en) * | 2020-06-26 | 2021-12-30 | Nvidia Corporation | Image generation using one or more neural networks |
| CN112862671A (en) * | 2021-02-09 | 2021-05-28 | 清华大学 | Video image editing and repairing method, device and storage medium |
| CN113658231A (en) * | 2021-07-07 | 2021-11-16 | 北京旷视科技有限公司 | Optical flow prediction method, device, electronic device and storage medium |
| CN113538275A (en) * | 2021-07-14 | 2021-10-22 | 华中科技大学 | A method and system for fruit occlusion recovery based on CycleGAN |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115118948B (en) | 2024-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11798132B2 (en) | Image inpainting method and apparatus, computer device, and storage medium | |
| CN110188760B (en) | Image processing model training method, image processing method and electronic equipment | |
| KR101986592B1 (en) | Recognition method of license plate number using anchor box and cnn and apparatus using thereof | |
| CN113689540B (en) | Object reconstruction method and device based on RGB video | |
| CN111260666B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| CN109583345B (en) | Road recognition method, device, computer device and computer readable storage medium | |
| CN109413411B (en) | Black screen identification method and device of monitoring line and server | |
| JP7092615B2 (en) | Shadow detector, shadow detection method, shadow detection program, learning device, learning method, and learning program | |
| CN111553923B (en) | Image processing method, electronic equipment and computer readable storage medium | |
| CN113281780B (en) | Method and device for marking image data and electronic equipment | |
| CN107273895B (en) | Method for recognizing and translating real-time text of video stream of head-mounted intelligent device | |
| CN111950610B (en) | A detection method of weak and small human objects based on accurate scale matching | |
| CN110807362A (en) | Image detection method and device and computer readable storage medium | |
| CN109472193A (en) | Method for detecting human face and device | |
| CN113052170B (en) | Small target license plate recognition method under unconstrained scene | |
| CN107563299B (en) | Pedestrian detection method using RecNN to fuse context information | |
| CN113095239B (en) | Key frame extraction method, terminal and computer readable storage medium | |
| CN110942456B (en) | Tamper image detection method, device, equipment and storage medium | |
| KR102546631B1 (en) | Apparatus for video data argumentation and method for the same | |
| CN112257729A (en) | Image recognition method, device, equipment and storage medium | |
| CN112949453B (en) | Training method of smoke and fire detection model, smoke and fire detection method and equipment | |
| CN115965934A (en) | Parking space detection method and device | |
| CN108229281B (en) | Neural network generation method, face detection device and electronic equipment | |
| US12026929B2 (en) | Method for using target pixels to remove objects from texture | |
| JP3232502B2 (en) | Fog monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |