CN113453017B - Video processing method, device, equipment and computer program product - Google Patents

Abstract

The invention discloses a video processing method, which comprises the following steps: performing lens segmentation operation on a key frame of a video to be processed to obtain a plurality of image groups; acquiring color difference, and respectively determining each reference frame and a frame to be processed based on the first frame; respectively acquiring color value corresponding relations between the part areas of the frames to be processed in each image group and the color value ranges; and acquiring a first low color gamut image frame corresponding to each frame to be processed, and determining a compressed video corresponding to the video to be processed based on the color difference, the first low color gamut image frame, the color value corresponding relation and the reference frame. The invention also discloses a video processing device, equipment and a computer program product. According to the method, the structure of the key frame is disassembled, the color value ranges of different part areas are compressed, the compression ratio of intraframe compression is improved, the selection of the reference frame is carried out according to the similarity of each frame in each lens, and the accuracy of the reference frame is improved.

Description

Video processing method, apparatus, apparatus and computer program product

technical field

The present invention relates to the field of video processing, and in particular, to a video processing method, apparatus, device and computer program product.

Background technique

With the development of mobile devices such as smart phones, tablet computers, and e-readers, multimedia data is widely used on mobile terminals. Since the network environment where the mobile terminal is located is volatile and complex, the available bandwidth is easily limited. Therefore, when the mobile terminal needs to download or upload a video file with a large amount of data to the network side, the video file usually needs to be compressed.

Generally, video compression includes intra-frame compression and inter-frame compression. Intra-frame compression techniques compress individual frames, often referred to as I-frames or keyframes. Interframe compression techniques compress frames with reference to previous or subsequent frames, which are commonly referred to as predicted frames, P-frames, or B-frames. Existing video compression adopts intra-frame compression of key frames in a GOP (Group of Pictures, group of pictures), and video compression is performed by means of inter-frame prediction of image residuals. However, the existing intra-frame compression methods do not process different regions of key frames in the same shot, resulting in low compression of intra-frame compression.

The above content is only used to assist the understanding of the technical solutions of the present invention, and does not mean that the above content is the prior art.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a video processing method, device, equipment and computer program product, aiming at solving the technical problem of low compression ratio of key frame intra-frame compression in the existing video compression.

In order to achieve the above object, the present invention provides a video processing method, and the video processing method includes the following steps:

Performing a shot segmentation operation on the key frames of the video to be processed to obtain multiple image groups, wherein each frame in the image group belongs to the same shot;

Obtain the color difference of people and objects in each image group, determine the target similarity of each frame based on the similarity between two adjacent frames in the image group, respectively determine that the target similarity in each image group is greater than or equal to a preset threshold The first frame, based on the first frame to determine the respective reference frame and the frame to be processed;

Perform a structural disassembly operation on the to-be-processed frame to obtain the color value range corresponding to the part area in the to-be-processed frame respectively, and obtain the color value correspondence between the part area of the to-be-processed frame and the color value range in each image group respectively relation;

Acquire a first low-color gamut image frame corresponding to each of the frames to be processed, and determine the to-be-processed image frame based on the color difference, the first low-color gamut image frame, the color value correspondence, and the reference frame The compressed video corresponding to the video.

Further, the step of performing a structure disassembly operation on the to-be-processed frame to obtain the color value ranges corresponding to the part regions in the to-be-processed frame respectively includes:

Based on the preset rgb color partitions, the to-be-processed frames are respectively mapped, to obtain the mapped to-be-processed frames, to obtain the second low-color gamut image frames corresponding to the mapped to-be-processed frames, and to determine each first image frame based on the color value continuity rule. A dividing line in a low-color gamut image frame, and based on the dividing line, respectively determine the part regions in the frame to be processed, and obtain the color value range corresponding to each part region based on the to-be-processed frame; or,

Recognize the image of the frame to be processed to obtain the part area in the frame to be processed, and obtain the color value range corresponding to each part area based on the frame to be processed, wherein the part area includes a person or object in the frame to be processed different parts of.

Further, the step of acquiring the first low color gamut image frame corresponding to each of the to-be-processed frames includes:

Convert the frame to be processed into a black and white image frame or a wireframe image frame to obtain the first low color gamut image frame; or,

Based on an image edge detection algorithm, the outline of the object in each frame to be processed is determined, and a Fourier transform operation is performed on the outline coordinates of the outline of the object to obtain the first low-color gamut image frame.

Further, the steps of determining the target similarity of each frame based on the similarity between two adjacent frames in the image group respectively include:

Based on the preset rgb color partition, map the image frames in the respective image groups to obtain the mapped image groups;

Obtaining the color histogram vector of each image frame in the image group after mapping, determining the similarity between two adjacent frames in each image group based on the color histogram vector, and determining the target similarity of each frame based on the description; or,

Perform a normalization operation on the mapped image group to obtain a normalized image group, obtain the pixel pairs corresponding to the differentiated pixel points between two adjacent image frames in the normalized image group, and Determine the pixel similarity between each pixel pair; based on the pixel similarity, determine the similarity between two adjacent frames in each image group, and determine the target similarity of each frame based on the similarity.

Further, the step of determining the compressed video corresponding to the to-be-processed video based on the color difference, the first low color gamut image frame, the color value correspondence and the reference frame includes:

Obtain the low color gamut picture corresponding to each of the frames to be processed and the color picture including the UV component, use the low color gamut picture as the input of the GAN model to be trained, and use the color picture as the output of the GAN model to be trained. training to obtain a pre-trained GAN model, wherein the GAN model to be trained is a model GAN corresponding to the style of the video to be processed;

If each image group has a second frame whose target similarity is less than the preset threshold, then based on the color difference, the pre-trained GAN model, the low color gamut picture, the second frame, the color value correspondence, and the reference frame, determine the compressed video corresponding to the to-be-processed video;

If each image group does not have a second frame whose target similarity is less than the preset threshold, then based on the color difference, the pre-trained GAN model, the low color gamut picture, the color value correspondence, and the reference frame, Determine the compressed video corresponding to the video to be processed.

Further, after the step of determining the compressed video corresponding to the to-be-processed video based on the color difference, the first low color gamut image frame, the color value correspondence and the reference frame, the video Processing methods also include:

Obtain the pre-trained GAN model corresponding to the compressed video, and input the video frame in the compressed video into the pre-trained GAN model for model training to obtain the video frame after training;

Based on the Y value of the reference frame in each image group of the compressed video, each image group in the trained video frame is processed to obtain a processed video frame;

Determine whether each image group corresponding to the processed video frame has a color difference image group based on the color difference and the color value correspondence in the compressed video;

If it does not exist, the processed video frame is used as the decoded target video.

Further, after the step of determining whether each image group of the processed video frame has a color difference image group, it also includes:

If there is, acquiring the first color distribution corresponding to the difference image frame in the color difference image group, and the second color distribution corresponding to the reference frame in the difference image group;

Matching processing is performed on the first color distribution based on the second color distribution to obtain the processed difference image frame, and the processed difference image frame is used to replace the difference image frame in the color difference image group, and the processed difference image frame is obtained. Color difference image group;

The target video is determined based on the processed group of color difference pictures and the group of normal pictures in the processed video frame.

In addition, in order to achieve the above object, the present invention also provides a video processing device, the video processing device comprising:

a segmentation module, configured to perform a shot segmentation operation on the key frames of the video to be processed to obtain multiple image groups, wherein each frame in the image group belongs to the same shot;

The first acquisition module is used to acquire the color difference of the characters and objects in each image group, determine the target similarity of each frame based on the similarity between two adjacent frames in the image group, respectively determine the target similarity in each image group a first frame whose degree is greater than or equal to a preset threshold, and each reference frame and a frame to be processed are respectively determined based on the first frame;

A dismantling module, configured to perform a structural dismantling operation on the to-be-processed frame, so as to obtain the color value range corresponding to the part area in the to-be-processed frame, respectively, and to obtain the part area and color value range of the to-be-processed frame in each image group respectively The color value correspondence between;

The second acquisition module is configured to acquire the first low color gamut image frame corresponding to each of the frames to be processed, based on the color difference, the first low color gamut image frame, the color value correspondence and the reference frame, and determine the compressed video corresponding to the video to be processed.

In addition, in order to achieve the above object, the present invention also provides a video processing device, the video processing device includes: a memory, a processor, and a video processing program stored in the memory and running on the processor. When the video processing program is executed by the processor, the steps of the aforementioned video processing method are implemented.

In addition, in order to achieve the above object, the present invention also provides a computer program product, including a computer program, which implements the steps of the aforementioned video processing method when the computer program is executed by a processor.

The present invention obtains a plurality of image groups by performing lens segmentation on the key frames of the video to be processed, and then obtains the color difference of the characters and objects in each image group, and determines the difference based on the similarity between two adjacent frames in the image group respectively. For the target similarity of each frame, determine the first frame of each image group whose target similarity is greater than or equal to a preset threshold, and determine each reference frame and to-be-processed frame based on the first frame; and then determine the to-be-processed frame. The frame is subjected to a structural disassembly operation to obtain the color value range corresponding to the part area in the frame to be processed, and obtain the color value correspondence between the part area of the frame to be processed and the color value range in each image group; then obtain each image group. The first low color gamut image frame corresponding to the to-be-processed frame, based on the color difference, the first low-color gamut image frame, the color value correspondence, and the reference frame, determine that the to-be-processed video corresponds to Compressed video, by dismantling the structure of key frames, compressing the color value range of different parts and regions, and improving the compression ratio of intra-frame compression. At the same time, the reference frame is selected according to the similarity of each frame in each shot, which improves the accuracy of the reference frame.

Description of drawings

1 is a schematic structural diagram of a video processing device in a hardware operating environment involved in an embodiment of the present invention;

2 is a schematic flowchart of a first embodiment of a video processing method according to the present invention;

3 is a schematic diagram of disassembling the structure of a frame to be processed in an embodiment of the video processing method of the present invention;

FIG. 4 is a schematic diagram of functional modules of an embodiment of a video processing apparatus according to the present invention.

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

As shown in FIG. 1 , FIG. 1 is a schematic structural diagram of a video processing device in a hardware operating environment involved in an embodiment of the present invention.

The video processing device in the embodiment of the present invention may be a PC, or a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer III) player, and an MP4 (Moving Picture Experts Group Audio Layer III). Picture Experts Group Audio Layer IV, moving image expert compression standard audio layer 4) Players, portable computers and other portable terminal devices with display functions.

As shown in FIG. 1 , the video processing device may include: a processor 1001 , such as a CPU, a network interface 1004 , a user interface 1003 , a memory 1005 , and a communication bus 1002 . Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (eg, a WI-FI interface). The memory 1005 may be high-speed RAM memory, or may be non-volatile memory, such as disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Optionally, the video processing device may further include a camera, an RF (Radio Frequency, radio frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors such as light sensors, motion sensors and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor can turn off the display screen and/or when the video processing device is moved to the ear. or backlight. As a kind of motion sensor, the gravitational acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that recognize the posture of mobile terminals (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, video processing equipment can also be equipped with gyroscopes, barometers, hygrometers, thermometers, infrared sensors and other sensors. This will not be repeated here.

Those skilled in the art can understand that the terminal structure shown in FIG. 1 does not constitute a limitation on the video processing device, and may include more or less components than the one shown, or combine some components, or arrange different components.

As shown in FIG. 1 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a video processing program.

In the terminal shown in FIG. 1 , the network interface 1004 is mainly used to connect to the background server and perform data communication with the background server; the user interface 1003 is mainly used to connect to the client (client) and perform data communication with the client; and the processor 1001 can be used to call a video processing program stored in memory 1005 .

In this embodiment, the video processing device includes: a memory 1005, a processor 1001, and a video processing program stored on the memory 1005 and running on the processor 1001, wherein the processor 1001 calls the memory 1005 to store the When the video processing program is created, the steps of the video processing methods in the following embodiments are executed.

The present invention also provides a video processing method. Referring to FIG. 2 , FIG. 2 is a schematic flowchart of the first embodiment of the video processing method of the present invention.

In this embodiment, the video processing method includes the following steps:

Step S101, performing a shot segmentation operation on the key frame of the video to be processed to obtain a plurality of image groups, wherein each frame in the image group belongs to the same shot;

In this embodiment, when the video to be processed is acquired, the shot segmentation technology is used to perform shot segmentation on the key frames of the video to be processed, so as to obtain multiple image groups, so that each frame in each image group belongs to the same shot, Specifically, existing lens segmentation techniques such as feature comparison and pixel comparison can be used. When the image difference between the two frames before and after the key frame is greater than the preset difference value, it is considered that there is a change in the lens, and then the two frames before and after are regarded as the image group. For the divided frame, the two frames before and after belong to two image groups respectively, that is, all frames between the previous frame and the following frame of the previous divided frame are regarded as an image group, and the image group includes the previous frame and the latter frame of the previous divided frame.

Step S102, obtain the color difference of the characters and objects in each image group, determine the target similarity of each frame based on the similarity between two adjacent frames in the image group, and determine that the target similarity in each image group is greater than or equal to the predetermined value. The first frame of the threshold is set, and each reference frame and the frame to be processed are respectively determined based on the first frame;

In this embodiment, an image recognition algorithm is first used to perform image recognition on each frame of each image group to determine the person and object in the frame of each image group, and to obtain the maximum area of the person and the minimum size of the person in each image group. Area, the maximum area of the object, and the minimum area of the object, where the maximum and minimum areas of the characters include the maximum and minimum areas of each character in the key frame of the image group, and the maximum and minimum areas of the object include the key of the image group. The maximum area and the minimum area of each object in the frame. For each image group, obtain the maximum frame of the character to which the maximum area of each character belongs and the minimum frame of the character to which the minimum area belongs, and the maximum frame and minimum frame of the object to which the maximum area of the object belongs. The minimum frame of the object to which the area belongs. Then, based on the pixel value of the area corresponding to the person in the largest frame of the person and the pixel value of the area corresponding to the person in the smallest frame of the person, the color difference of the person is calculated. The object color difference, and the character color difference and the object color difference are regarded as the color difference.

Next, the similarity between two adjacent frames in the image group is obtained. Specifically, the existing similarity may be used, and the target similarity of each frame in the image group is determined based on the similarity. For example, if a certain frame is an image If the start frame or end frame of the group, the similarity between the start frame and the next frame is used as the target similarity of the start frame, and the similarity between the end frame and the previous frame is used as the target similarity of the end frame. For an intermediate frame, the target similarity is determined according to the similarity between the frame and the next frame and the similarity with the previous frame. For example, the target similarity is the average of the similarity with the next frame and the similarity with the previous frame.

When the target similarity of each frame is obtained, determine the first frame of each image group whose target similarity is greater than or equal to the preset threshold. Specifically, for each image group, determine whether the target similarity of each frame is Greater than or equal to the preset threshold, take the frame with the target similarity greater than or equal to the preset threshold as the first frame, and determine each reference frame and the frame to be processed based on the first frame, wherein, any one of the first frames can be selected. One frame is used as the reference frame, or the frame with the highest target similarity in the first frame is used as the reference frame, and other frames in the first frame except the reference frame are used as the frames to be processed.

It should be noted that, for each frame in the image group of the same shot, if the target similarity of the frame is greater than or equal to the preset threshold, then compared with the previous frame and the next frame, the object or person in the frame is The color of the frame is only changed due to the environment, light, etc., which belongs to the normal range, so this frame is regarded as the first frame. Of course, if the target similarity of the frame is less than the preset threshold, the object or character in the frame changes color due to its own reasons. For example, the light color of the traffic light changes. When compressing the frame, do not compress the frame. .

Step S103, performing a structural disassembly operation on the frame to be processed, to obtain the color value range corresponding to the part area in the frame to be processed, respectively, and to obtain the part area and the color value range of the frame to be processed in each image group. Correspondence of color value;

In this embodiment, after the to-be-processed frames of each image group are acquired, a structure disassembly operation is performed on the to-be-processed frames of each image group, and a structure disassembly operation is performed on the to-be-processed frames, so as to obtain the The color value range corresponding to the part area of in which the part area in the frame to be processed includes various parts of the person and the object or multiple areas with discontinuous color values, and then the color value is determined based on the rgb value of the pixel in each part area. Range, the color value range can be the minimum rgb value of the pixel in the part area to the rgb maximum value of the pixel in the part area, and record the color value correspondence between the part area of the frame to be processed and the color value range in each image group .

Step S104: Obtain the first low color gamut image frame corresponding to each of the frames to be processed, and determine the color difference based on the color difference, the first low color gamut image frame, the color value correspondence, and the reference frame. The compressed video corresponding to the video to be processed.

In this embodiment, after the color value correspondence is determined, the first low color gamut image frame corresponding to each of the to-be-processed frames is obtained, for example, each to-be-processed frame is converted into a black and white picture or a wireframe to obtain the The corresponding first low color gamut image frame, and according to the color difference, the first low color gamut image frame, the color value correspondence and the reference frame, determine the compressed video corresponding to the to-be-processed video, that is, the color difference, the first low-color gamut image frame, the color value correspondence, and the reference frame, and perform video encapsulation in the order of the image groups to obtain a compressed video,

The color value correspondence can be encapsulated in the video format of the compressed video. For example, if the compressed video is only used locally, the color value correspondence can be encapsulated in the video header. If the compressed video is used for network transmission, it can be The color value corresponding relationship of the frame to be processed in each image group is obtained according to the color value corresponding relationship, and the color value corresponding relationship of the to-be-processed frame in the image group is encapsulated before the image group.

The video processing method proposed in this embodiment obtains multiple image groups by performing lens segmentation on the key frames of the video to be processed, and then acquires the color differences of the characters and objects in each image group, based on the two adjacent quantities in the image groups. The similarity between the frames determines the target similarity of each frame, respectively determines the first frame whose target similarity is greater than or equal to a preset threshold in each image group, and determines each reference frame and to-be-processed frame based on the first frame. Then carry out the structure disassembly operation to the frame to be processed, to obtain the color value range corresponding to the part area in the frame to be processed, respectively, to obtain the color value range between the part area of the frame to be processed and the color value range in each image group. value correspondence; then obtain the first low color gamut image frame corresponding to each of the to-be-processed frames, based on the color difference, the first low color gamut image frame, the color value correspondence and the reference frame, Determine the compressed video corresponding to the to-be-processed video, and compress the color value ranges of different parts and regions by disassembling the key frame structure, thereby improving the compression ratio of the intra-frame compression. At the same time, the reference frame is selected according to the similarity of each frame in each shot, which improves the accuracy of the reference frame.

Based on the first embodiment, a second embodiment of the video processing method of the present invention is proposed. In this embodiment, step S103 includes:

Step S201, respectively map the to-be-processed frame based on the preset rgb color partition to obtain the mapped to-be-processed frame, obtain the second low-gamut image frame corresponding to the mapped to-be-processed frame, based on the color value continuity rule Determining a dividing line in each of the second low color gamut image frames, and respectively determining a part area in a frame to be processed based on the dividing line, and obtaining a color value range corresponding to each part area based on the frame to be processed; or,

Step S202, perform an image recognition operation on the to-be-processed frame to obtain the part area in the to-be-processed frame, and obtain the color value range corresponding to each part area based on the to-be-processed frame, wherein the part area includes the part area in the to-be-processed frame. different parts of a person or object.

In one way, in order to reduce the amount of calculation, the image frames in the respective image groups are mapped based on the preset rgb color partitions to obtain the mapped image groups. For example, the preset rgb color partitions can be set to 8. Or 16 partitions, of course, more partitions can be set. When the preset rgb color partition includes 8 partitions, the color value ranges of r, g, and b are: 0～31, 32～63, 64～ 95, 95~127, 128~159, 160~191, 192~223, 224~255.

Then, obtain the second low color gamut image frame corresponding to the mapped frame to be processed, that is, convert the mapped frame to be processed into a black and white image (or grayscale image), and determine each second low color gamut based on the color value continuity rule The dividing line in the image frame, and based on the dividing line, the part regions in the frame to be processed are respectively determined, and according to the color value continuity rule, it is determined that the difference between adjacent pixels in the second low color gamut image frame is large and different When the chromatized color is continuous, the continuous pixels are considered to be the dividing lines of different parts. Specifically, the pixel similarity between adjacent pixels in each second low-color gamut image frame can be calculated, and when the pixel similarity between adjacent pixels is less than the preset pixel similarity, determine the difference between the adjacent pixels. The difference between the two is larger, and the adjacent pixel is regarded as a differentiated pixel, or the average of the pixel similarity between the current pixel and other adjacent pixels is used as the current similarity of the current pixel, and the second pixel is regarded as the difference. In the low color gamut image frame, the pixels whose current similarity is less than the preset pixel similarity are regarded as differentiated pixels. If the differentiated pixels show continuity, the connection between the consecutive differentiated pixels is regarded as the corresponding first pixel. The dividing line in a low-gamut image frame. Then, based on the demarcation line, the part area in the frame to be processed is respectively determined, and the part area is each area enclosed by the demarcation line in the second low color gamut image frame, and the color value corresponding to each part area is obtained based on the frame to be processed scope.

Referring to Fig. 3, in Fig. 3, there are 4 color values, among which 3 colors are compared, and the difference is relatively small, while the other 1 color is more differentiated from the other 3, and it can be considered that the upper, left and right three The color values form the dividing line.

In another way, the image recognition operation is performed on the frame to be processed to obtain the part area in the frame to be processed, and the part area includes different parts of the person or object in the frame to be processed, for example, for the person in the frame to be processed, The parts area includes the facial features, hair, clothes, shoes, decorations, etc. of the characters. Then, based on the to-be-processed frame, the color value range corresponding to each part region is acquired.

In the video processing method proposed in this embodiment, the image frames in the image groups are respectively mapped based on the preset rgb color partitions to obtain the mapped image groups, and the corresponding frames in the mapped image groups are obtained. For the second low color gamut image frame, based on the color value continuity rule, determine the dividing line in each second low color gamut image frame, and determine the part area in the frame to be processed based on the dividing line. Frame acquisition of the color value range corresponding to each part area; or, performing an image recognition operation on the to-be-processed frame image of each image group to obtain the part area in the to-be-processed frame, and based on the to-be-processed frame to acquire the color value range corresponding to each part area , wherein the part area includes different parts of the person or object in the frame to be processed, and by performing image recognition according to the second low-gamut image frame or the frame to be processed, the part area in the frame to be processed can be accurately obtained. The frame to be processed is accurately disassembled, and then the color value ranges of different parts and regions are compressed, which improves the compression ratio of intra-frame compression.

Based on the first embodiment, a third embodiment of the video processing method of the present invention is proposed. In this embodiment, step S104 includes:

Step S301, converting the frame to be processed into a black and white image frame or a wireframe image frame to obtain the first low color gamut image frame; or,

Step S302 , determine the contour of the object in each frame to be processed based on the image edge detection algorithm, and perform Fourier transform operation on the contour coordinates of the contour of the object to obtain the first low color gamut image frame.

In this embodiment, the frame to be processed may be directly converted into a black-and-white image frame or a wireframe image frame to obtain the first low-color gamut image frame, wherein the black-and-white image frame may be a binary image, or the previous implementation Grayscale image in the example.

Or, based on the image edge detection algorithm, determine the contour of the object in each frame to be processed, specifically use the existing image edge detection algorithm to detect the contour of the object in each frame to be processed, and then split the contour of the object into multiple points to obtain each The coordinates of the points are used to obtain the contour coordinates of the contour of the object, and a Fourier transform operation is performed on the contour coordinates to obtain the first low color gamut image frame.

In the video processing method proposed in this embodiment, the first low-color gamut image frame is obtained by converting the frame to be processed into a black-and-white image frame or a wireframe image frame; or, based on an image edge detection algorithm, each For the contour of the object in the frame to be processed, Fourier transform is performed on the contour coordinates of the contour of the object to obtain the first low-color gamut image frame, and the first low-color gamut image frame can be accurately obtained. Image frames further improve the compression ratio of intra-frame compression.

Based on the first embodiment, a fourth embodiment of the video processing method of the present invention is proposed. In this embodiment, step S102 includes:

Step S401, based on the preset rgb color partition, map the image frames in the respective image groups to obtain the mapped image groups;

Step S402, obtaining the color histogram vector of each image frame in the mapped image group, determining the similarity between two adjacent frames in each image group based on the color histogram vector, and determining the target of each frame based on the described color histogram vector. similarity; or,

Step S403, perform a normalization operation on the mapped image group to obtain a normalized image group, and obtain the pixels corresponding to the differentiated pixel points between two adjacent image frames in the normalized image group pair, and determine the pixel similarity between each pixel pair; determine the similarity between two adjacent frames in each image group based on the pixel similarity, and determine the target similarity of each frame based on the pixel similarity.

In this embodiment, in order to reduce the amount of calculation, the image frames in the respective image groups are mapped based on the preset rgb color partitions, so as to obtain the mapped image groups. For example, the preset rgb color partitions can be set to 8 Or 16 partitions, of course, more partitions can be set. When the preset rgb color partition includes 8 partitions, the color value ranges of r, g, and b are: 0～31, 32～63, 64～ 95, 95~127, 128~159, 160~191, 192~223, 224~255.

In one embodiment, the color histogram vector of each image frame in the mapped image group is obtained, that is, an existing color histogram vector algorithm is used to obtain the color histogram vector, and each image group is determined based on the color histogram vector The similarity between two adjacent frames in the middle, specifically, the cosine value of the color histogram vector corresponding to the two adjacent frames is used as the similarity.

In another embodiment, a normalization operation is performed on the mapped image group to obtain a normalized image group, and the differentiated pixels between two adjacent image frames in the normalized image group are obtained Point the corresponding pixel pair, determine the pixel similarity between each pixel pair, specifically, obtain the color partition list of two adjacent image frames, and find the differentiated pixels (pixels with different rgb values) according to the uniform sorting. , to obtain the pixel pairs corresponding to the differentiated pixel points, and then calculate the pixel similarity between each pixel pair through the cosine of the rgb space vector. Then, the similarity between two adjacent frames in each image group is determined based on the pixel similarity, specifically, the similarity=(pixel similarity+number of differentiated pixels*1)/total number of pixels.

The video processing method proposed in this embodiment obtains the color histogram vector of each image frame in the mapped image group, and determines the similarity between two adjacent frames in each image group based on the color histogram vector, Determine the target similarity of each frame based on the above; or, perform a normalization operation on the mapped image group to obtain a normalized image group, and obtain two adjacent image frames in the normalized image group The pixel pairs corresponding to the differentiated pixel points between them are determined, and the pixel similarity between each pixel pair is determined; based on the pixel similarity, the similarity between two adjacent frames in each image group is determined, based on the pixel similarity. Determining the target similarity of each frame can accurately obtain the target similarity according to the color histogram vector or the pixel pairs corresponding to the differentiated pixel points, thereby improving the accuracy of the first frame and further improving the video compression ratio.

Based on the foregoing embodiments, a fifth embodiment of the video processing method of the present invention is proposed. In this embodiment, step S104 includes:

Step S501, obtain the low color gamut picture corresponding to each frame to be processed and the color picture including the UV component, use the low color gamut picture as the input of the GAN model to be trained, and use the color picture as the input of the GAN model to be trained. The output carries out model training to obtain a pre-trained GAN model, wherein the to-be-trained GAN model is the model GAN corresponding to the style of the to-be-processed video;

Step S502, if each image group has a second frame whose target similarity is less than a preset threshold, then based on the color difference, the pre-trained GAN model, the low color gamut picture, the second frame, and the color value corresponding relationship and the reference frame, to determine the compressed video corresponding to the video to be processed;

Step S503, if each image group does not have a second frame whose target similarity is less than a preset threshold, then based on the color difference, the pre-trained GAN model, the low color gamut picture, the color value correspondence and the With reference to the frame, the compressed video corresponding to the video to be processed is determined.

In this embodiment, after the first low-color gamut image frame is acquired, the corresponding GAN model to be trained is determined according to the style of the video to be processed, and the low-color gamut image corresponding to each of the to-be-processed frames and the image including the UV component are acquired. For color pictures, specifically, the RGB of the frame to be processed is converted into the color space YUV (wherein Y represents the brightness, that is, the gray value, and U and V represent the chromaticity) by using the existing algorithm. Then, the low color gamut image is used as the input of the GAN model to be trained, and the color image is used as the output of the GAN model to be trained for model training to obtain a pre-trained GAN model.

Next, determine whether each image group has a second frame whose target similarity is less than the preset threshold. If there is, the second frame is used as a reserved frame, the second frame is not encoded, and the GAN model based on color difference and pre-training is used. , low color gamut picture, second frame, color value correspondence and reference frame, determine the compressed video; that is, color difference, pre-trained GAN model, low color gamut picture, second frame, color value correspondence and reference frame, according to Video encapsulation is performed in the sequence of GOPs to obtain compressed video.

If it does not exist, the color difference, the pre-trained GAN model, the low color gamut picture, the color value correspondence and the reference frame are encapsulated in the order of the image group to obtain the compressed video.

In this embodiment, if the compressed video is used locally, only the model number corresponding to the pre-trained GAN model is encapsulated during video encapsulation, otherwise the pre-trained GAN model can be directly encapsulated.

In the video processing method proposed in this embodiment, the low-color gamut pictures corresponding to the frames to be processed and the color pictures including UV components are obtained, the low-color gamut pictures are used as the input of the GAN model to be trained, and the color pictures are used as the input of the GAN model to be trained. The picture is used as the output of the GAN model to be trained, and model training is performed to obtain a pre-trained GAN model, wherein the GAN model to be trained is the model GAN corresponding to the style of the video to be processed; then if each image group has a target similarity For the second frame smaller than the preset threshold, the to-be-processed frame is determined based on the color difference, the pre-trained GAN model, the low color gamut picture, the second frame, the color value correspondence, and the reference frame The compressed video corresponding to the video; if there is no second frame whose target similarity is less than the preset threshold in each image group, based on the color difference, the pre-trained GAN model, the low color gamut picture, and the color value correspondence and the reference frame, determine the compressed video corresponding to the to-be-processed video, and encapsulate the information of the pre-trained GAN model into the compressed video, so as to facilitate subsequent decoding and coloring of the compressed video, and at the same time by retaining the second frame to To ensure the authenticity of the second frame, to avoid serious distortion of the decoded image by encoding the second frame.

Based on the fifth embodiment, a sixth embodiment of the video processing method of the present invention is proposed. In this embodiment, after step S104, the video processing method further includes:

Step S601, obtaining a pre-trained GAN model corresponding to the compressed video, and inputting the video frame in the compressed video into the pre-trained GAN model for model training to obtain a trained video frame;

Step S602, processing each image group in the video frame after training based on the Y value of the reference frame in each image group of the compressed video to obtain a processed video frame;

Step S603, based on the color difference in the compressed video and the color value correspondence, determine whether each image group corresponding to the processed video frame has a color difference image group;

Step S604, if it does not exist, use the processed video frame as the decoded target video.

In this embodiment, when the compressed video needs to be decoded, first obtain the pre-trained GAN model corresponding to the compressed video, for example, obtain the pre-trained GAN model locally by using the model number in the compressed video, or obtain the pre-trained GAN model in the compressed video Pretrained GAN model. And input the video frames in the compressed video into the pre-trained GAN model for model training to obtain the trained video frames. Among them, the reference frame and the second frame do not need model training.

Next, obtain reference frames in each image group of the compressed video, and obtain the Y value (brightness value) of each reference frame, and process each image group in the trained video frame based on each Y value to obtain the processed For any trained image group in the trained video frame, replace the Y value of the video frame in the trained image group with the Y value of the corresponding reference frame to obtain the trained video frame.

Then, based on the color difference and color value correspondence in the compressed video, determine whether each image group corresponding to the processed video frame has a color difference image group; based on the same processing method as the color difference in the compressed video, obtain the processed image The current color difference of the characters and objects in each image group corresponding to the video frame, and then determine whether the difference between the current color difference and the color difference of each character and object is less than the preset color difference value, if the difference value is greater than or equal to the preset color difference Color difference value, then determine whether the color value of the part region of the processed video frame is within the corresponding color value range based on the color value correspondence, and if so, determine whether there is a color difference image in each image group corresponding to the processed video frame group, and then use the processed video frame as the decoded target video.

Further, in an embodiment, after step S604, it further includes:

Step a, if there is, then obtain the first color distribution corresponding to the difference image frame in the color difference image group, and the second color distribution corresponding to the reference frame in the difference image frame;

Step b, performing matching processing on the first color distribution based on the second color distribution to obtain a processed difference image frame, and replacing the difference image frame in the color difference image group with the processed difference image frame, to obtain: The processed color difference image group;

Step c, determining the target video based on the processed color difference image group and the processed normal image group in the video frame.

If there is, use the video frame with the current color difference whose difference value is less than the preset color difference value in the processed video frame as the difference image frame, and/or set the color value of the part area in the processed video frame in the corresponding color The video frames outside the value range are used as difference image frames, and the reference frame is determined according to the difference image group to which the difference image frame belongs, and the first color distribution corresponding to the difference image frame in the color difference image group is obtained, and the difference image group in the difference image group is obtained. The second color distribution corresponding to the reference frame, wherein the first color distribution is the color distribution of the pixels corresponding to the different characters, objects or parts, and the second color distribution is the pixels corresponding to the different characters, objects or parts in the reference frame color distribution.

Then, matching processing is performed on the first color distribution based on the second color distribution to obtain a processed difference image frame. Specifically, each pixel in the second color distribution is matched with each pixel in the first color distribution , adjust the pixel value of each pixel in the first color distribution to the pixel value of each pixel in the second color distribution, or set a certain error tolerance range, first obtain each pixel in the second color distribution The pixel value of each pixel in the first color distribution is adjusted to the closest pixel value in the corresponding pixel adjustment range.

For example, the pixel values of each pixel in the second color distribution are: #8F0000, #D70000, #F00000, and the pixel values of each pixel in the first color distribution are: #3F0000, #D80000, #F00000, The ratio of the pixel adjustment range is 10%. If #3F0000 exceeds #8F0000, #3F0000 needs to match the nearest color value #8F0000. The similarity between #D70000 and #D80000 is high. Within the fault tolerance range, #F00000 and #F00000 are the same as Consistent.

Finally, the difference image frame in the color difference image group is replaced by the processed difference image frame, and the processed color difference image group is obtained, and based on the processed color difference image group and the normal image group in the processed video frame, determine the target video

In the video processing method proposed in this embodiment, a pre-trained GAN model is obtained based on the model number in the compressed video, and the video frames in the compressed video are input into the pre-trained GAN model for model training, so as to obtain a post-training GAN model. Then, based on the Y value of the reference frame in each image group in the compressed video, each image group in the video frame after training is processed to obtain the processed video frame; then obtain the processed video frame. The second color difference of the person or object in each image group, based on the color difference and the second color difference, determine whether each image group of the processed video frame has a color difference image group; As the decoded target video, the frame can be accurately decoded according to the compressed video to obtain the target video.

The present invention also provides a video processing device, referring to FIG. 4 , the video processing device includes:

A segmentation module 10, configured to perform a shot segmentation operation on the key frames of the video to be processed, so as to obtain a plurality of image groups, wherein each frame in the image group belongs to the same shot;

The first obtaining module 20 is used to obtain the color difference of characters and objects in each image group, determine the target similarity of each frame based on the similarity between two adjacent frames in the image group, and determine the target similarity in each image group respectively. For the first frame whose similarity is greater than or equal to a preset threshold, determine the respective reference frame and the frame to be processed based on the first frame;

The disassembly module 30 is configured to perform a structural disassembly operation on the to-be-processed frame, so as to obtain the color value range corresponding to the part area in the to-be-processed frame, respectively, to obtain the part area and color value of the to-be-processed frame in each image group. The color value correspondence between the ranges;

The second obtaining module 40 is configured to obtain the first low color gamut image frame corresponding to each of the frames to be processed, based on the color difference, the first low color gamut image frame, the color value correspondence and the With reference to the frame, the compressed video corresponding to the video to be processed is determined.

For the methods executed by the above program units, reference may be made to the various embodiments of the video processing method of the present invention, which will not be repeated here.

The present invention also provides a computer-readable storage medium.

A video processing program is stored on the computer-readable storage medium of the present invention, and when the video processing program is executed by the processor, the steps of the above-mentioned video processing method are implemented.

For the method implemented when the video processing program running on the processor is executed, reference may be made to the various embodiments of the video processing method of the present invention, which will not be repeated here.

In addition, an embodiment of the present invention also provides a computer program product, the computer program product includes a video processing program, and when the video processing program is executed by a processor, implements the steps of the video processing method as described above.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or system. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system that includes the element.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on such understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM) as described above. , magnetic disk, optical disk), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields , are similarly included in the scope of patent protection of the present invention.

Claims (10)

1. A video processing method, characterized in that the video processing method comprises the steps of:

performing shot segmentation operation on key frames of a video to be processed to obtain a plurality of image groups, wherein each frame in the image groups belongs to the same shot;

acquiring color differences corresponding to each image group, determining target similarity of each frame based on the similarity between two adjacent frames in the image group, determining a first frame with the target similarity being larger than or equal to a preset threshold value in each image group, determining a reference frame and a frame to be processed based on the first frame, wherein the color differences comprise character color differences and object color differences, determining the character color differences for each image group based on a character maximum frame to which the maximum area of each character in the image group belongs and a character minimum frame to which the minimum area belongs, and determining the object color differences based on an object maximum frame to which the maximum area of each object belongs and an object minimum frame to which the minimum area belongs;

performing structure disassembly operation on the frame to be processed to respectively obtain color value ranges corresponding to part regions in the frame to be processed, and respectively obtaining color value corresponding relations between the part regions and the color value ranges of the frame to be processed in each image group, wherein the part regions comprise all parts of people and objects, or the part regions comprise a plurality of regions with discontinuous color values in the frame to be processed;

acquiring a first low color gamut image frame corresponding to each frame to be processed, and determining a compressed video corresponding to the video to be processed based on the color difference, the first low color gamut image frame, the color value corresponding relationship and the reference frame, wherein the color difference, the first low color gamut image frame, the color value corresponding relationship and the reference frame are subjected to video encapsulation according to the sequence of an image group to obtain the compressed video, and the first low color gamut image frame is a black-and-white image frame or a wire frame image frame corresponding to the frame to be processed.

2. The video processing method according to claim 1, wherein the step of performing a structure decomposition operation on the frame to be processed to obtain color value ranges corresponding to the portion regions in the frame to be processed respectively comprises:

respectively mapping the to-be-processed frames based on preset rgb color partitions to obtain mapped to-be-processed frames, obtaining second low color gamut image frames corresponding to the mapped to-be-processed frames, determining boundary lines in each second low color gamut image frame based on a color value continuity rule, respectively determining part areas in the to-be-processed frames based on the boundary lines, and obtaining color value ranges corresponding to the part areas based on the to-be-processed frames, wherein the second low color gamut image frames are black-and-white images or grayscale images; or,

and identifying the frame image to be processed to obtain part areas in the frame to be processed, and acquiring color value ranges corresponding to the part areas based on the frame to be processed, wherein the part areas comprise different parts of people or objects in the frame to be processed.

3. The video processing method of claim 1, wherein the step of obtaining a first low color gamut image frame corresponding to each of the frames to be processed comprises:

converting the frame to be processed into a black-and-white image frame or a wire frame image frame to obtain the first low color gamut image frame; or,

determining an object contour in each frame to be processed based on an image edge detection algorithm, and performing Fourier transform operation on contour coordinates of the object contour to obtain the first low color gamut image frame.

4. The video processing method of claim 1, wherein the step of determining the object similarity of each frame based on the similarity between two adjacent frames in the image group comprises:

respectively mapping the image frames in each image group based on a preset rgb color partition to obtain a mapped image group;

acquiring a color histogram vector of each image frame in the mapped image group, determining the similarity between two adjacent frames in each image group based on the color histogram vector, and determining the target similarity of each frame based on the determined similarity; or,

normalizing the mapped image group to obtain a normalized image group, acquiring pixel pairs corresponding to a differentiation pixel point between two adjacent image frames in the normalized image group, and determining the pixel similarity between each pixel pair; and determining the similarity between two adjacent frames in each image group based on the pixel similarity, and determining the target similarity of each frame based on the similarity.

5. The video processing method according to any one of claims 1 to 4, wherein the step of determining the compressed video corresponding to the video to be processed based on the color difference, the first low color gamut image frame, the color value correspondence, and the reference frame comprises:

obtaining a low color gamut picture and a color picture comprising UV components corresponding to each frame to be processed, taking the low color gamut picture as the input of a GAN model to be trained, and taking the color picture as the output of the GAN model to be trained for model training to obtain a pre-trained GAN model, wherein the GAN model to be trained is a model GAN corresponding to the style of the video to be processed;

if a second frame with the target similarity smaller than a preset threshold exists in each image group, determining a compressed video corresponding to the video to be processed based on the color difference, a pre-trained GAN model, the low color gamut picture, the second frame, the color value corresponding relation and the reference frame;

and if the second frame with the target similarity smaller than the preset threshold value does not exist in each image group, determining the compressed video corresponding to the video to be processed based on the color difference, the pre-trained GAN model, the low color gamut picture, the color value corresponding relation and the reference frame.

6. The video processing method according to claim 5, wherein after the step of determining the compressed video corresponding to the video to be processed based on the color difference, the first low color gamut image frame, the color value correspondence, and the reference frame, the video processing method further comprises:

acquiring a pre-trained GAN model corresponding to a compressed video, and inputting a video frame in the compressed video into the pre-trained GAN model for model training to obtain a trained video frame;

processing each image group in the trained video frame based on the Y value of the reference frame in each image group of the compressed video to obtain a processed video frame;

determining whether each image group corresponding to the processed video frame has a color difference image group or not based on the color difference and the color value corresponding relation in the compressed video, and specifically, acquiring the current color difference of a person and the current color difference of an object in each image group corresponding to the processed video frame; if the difference value between the current color difference value of the person and the color difference value of the corresponding image group in the compressed video is larger than or equal to the preset color difference value, and the difference value between the current color difference value of the object and the color difference value of the corresponding image group in the compressed video is larger than or equal to the preset color difference value, determining whether the color value of the part area of the processed video frame is in the corresponding color value range or not based on the color value corresponding relation, wherein if the color value is in the corresponding color value range, determining that the color difference image group does not exist in each image group corresponding to the processed video frame;

and if the target video does not exist, the processed video frame is taken as the decoded target video.

7. The video processing method of claim 6, wherein the step of determining whether the color difference image group exists in each image group of the processed video frame further comprises:

if the color difference image group exists, acquiring a first color distribution corresponding to a difference image frame in the color difference image group and a second color distribution corresponding to a reference frame in the difference image group;

matching the first color distribution based on the second color distribution to obtain a processed difference image frame, and replacing the difference image frame in the color difference image group with the processed difference image frame to obtain a processed color difference image group;

and determining the target video based on the processed color difference image group and the processed normal image group in the video frame.

8. A video processing apparatus, characterized in that the video processing apparatus comprises:

the device comprises a segmentation module, a processing module and a processing module, wherein the segmentation module is used for carrying out lens segmentation operation on key frames of a video to be processed so as to obtain a plurality of image groups, and each frame in the image groups belongs to the same lens;

the first acquisition module is used for acquiring color differences corresponding to each image group, determining the target similarity of each frame based on the similarity between two adjacent frames in the image group, determining the first frame with the target similarity being larger than or equal to a preset threshold value in each image group, determining each reference frame and each frame to be processed based on the first frame, wherein the color differences comprise character color differences and object color differences, determining the character color differences for each image group based on the maximum character frame to which the maximum area of each character in the image group belongs and the minimum character frame to which the minimum area belongs, and determining the object color differences based on the maximum object frame to which the maximum area of each object belongs and the minimum object frame to which the minimum area belongs;

a disassembling module, configured to perform structure disassembling operation on the frame to be processed to obtain color value ranges corresponding to part regions in the frame to be processed, and obtain color value correspondence between the part regions and the color value ranges of the frame to be processed in each image group, where the part regions include parts of people and objects, or the part regions include multiple regions with discontinuous color values in the frame to be processed;

the second acquisition module is used for acquiring each first low color gamut image frame corresponding to the frame to be processed, and determining the compressed video corresponding to the video to be processed based on the color difference, the first low color gamut image frame, the color value corresponding relation and the reference frame, wherein the color difference, the first low color gamut image frame, the color value corresponding relation and the reference frame are subjected to video packaging according to the sequence of the image group to obtain the compressed video, and the first low color gamut image frame is a black-and-white image frame or a wire frame image frame corresponding to the frame to be processed.

9. A video processing apparatus, characterized in that the video processing apparatus comprises: memory, processor and video processing program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the video processing method according to any of claims 1 to 7.

10. A computer-readable storage medium, characterized in that a video processing program is stored thereon, which when executed by a processor implements the steps of the video processing method according to any one of claims 1 to 7.

Images