CN113706393B - Video enhancement method, device, equipment and storage medium - Google Patents

Abstract

The application is suitable for the technical field of image processing, and provides a video enhancement method, a video enhancement device, video enhancement equipment and a storage medium. The method comprises the steps of obtaining a target video frame in a video to be processed, and carrying out image enhancement processing on the target video frame to obtain a target enhanced image corresponding to the target video frame; determining an enhanced image corresponding to the video frame to be processed according to the target enhanced image; taking the video frame to be processed as a target video frame, taking an enhanced image corresponding to the video frame to be processed as a target enhanced image, and returning to execute the step of determining the enhanced image corresponding to the video frame to be processed according to the target enhanced image until the enhanced images respectively corresponding to the preset number of video frames in the video to be processed are obtained; and determining the enhanced video corresponding to the video to be processed according to the enhanced image corresponding to the video frame in the video to be processed.

Description

Video enhancement method, device, equipment and storage medium

Technical Field

The present application belongs to the field of image processing technology, and in particular, relates to a video enhancement method, apparatus, device, and storage medium.

Background

In video shooting, due to the influence of the quality of video shooting equipment and shooting environment, poor quality of pictures in the shot video can often occur, for example, the whole picture is dark, the contrast is low, and the like, so that the identifiability of an interested object in the video is low, and the application value of the video is greatly reduced. Thus, enhancement processing of video to selectively highlight features of objects of interest in the video has become a necessary means to improve video quality.

The existing video enhancement mode is usually realized based on an image enhancement technology, and the method does not consider the association relation among multiple frames of images in the video, so that phenomena such as brightness jitter and flickering easily occur in the enhanced video, and the video enhancement effect is unstable.

Disclosure of Invention

In view of the above, embodiments of the present application provide a video enhancement method, apparatus, device and storage medium, so as to solve the technical problem that luminance jitter is easy to occur in the video enhancement method in the prior art.

In a first aspect, an embodiment of the present application provides a video enhancement method, including:

acquiring a target video frame in a video to be processed;

performing image enhancement processing on the target video frame to obtain a target enhanced image corresponding to the target video frame;

Determining an enhanced image corresponding to the video frame to be processed according to the target enhanced image; the video frame to be processed is determined according to the target video frame and is subjected to image enhancement processing;

Taking the video frame to be processed as a target video frame, taking an enhanced image corresponding to the video frame to be processed as a target enhanced image, and returning to execute the step of determining the enhanced image corresponding to the video frame to be processed according to the target enhanced image until the enhanced images respectively corresponding to the preset number of video frames in the video to be processed are obtained;

and determining the enhanced video corresponding to the video to be processed according to the enhanced image corresponding to the video frame in the video to be processed.

In a possible implementation manner of the first aspect, the video frame to be processed is a video frame adjacent to the target video frame.

In a possible implementation manner of the first aspect, performing image enhancement processing on a target video frame to obtain a target enhanced image corresponding to the target video frame, including:

acquiring a brightness channel of a target video frame;

determining a plurality of illumination components of the target video frame according to the brightness channel of the target video frame;

Determining a fusion illumination component corresponding to the target video frame according to the illumination components of the target video frame;

And processing the target video frame according to the fusion illumination component to generate a target enhanced image corresponding to the target video frame.

In a possible implementation manner of the first aspect, determining a plurality of illumination components of the target video frame according to a luminance channel of the target video frame includes:

performing low-pass filtering treatment on the brightness channel to obtain a first illumination component;

and performing a variation operation on the brightness channel to obtain a second illumination component.

In a possible implementation manner of the first aspect, determining, according to a plurality of illumination components of the target video frame, a fused illumination component corresponding to the target video frame includes:

Determining a first gradient of the first illumination component and a second gradient of the second illumination component;

and based on the first gradient and the second gradient, carrying out fusion processing on the first illumination component and the second illumination component to obtain a fusion illumination component.

In a possible implementation manner of the first aspect, the processing the target video frame according to the fused illumination component, to generate a target enhanced image corresponding to the target video frame, includes:

Determining a reflection component of the luminance channel according to the fused illumination component and the luminance channel;

Correcting the reflection component to generate a corrected reflection component;

Nonlinear stretching is carried out on the fusion illumination component, and a stretched fusion illumination component is generated;

obtaining an enhanced brightness channel according to the stretched fusion illumination component and the corrected reflection component;

And processing the target video frame according to the enhanced brightness channel to generate a target enhanced image corresponding to the target video frame.

In a possible implementation manner of the first aspect, the processing the target video frame according to the enhanced brightness channel, to generate a target enhanced image corresponding to the target video frame, includes:

Acquiring a saturation channel and a chromaticity channel of a target video frame;

Generating an RGB color image according to the enhanced brightness channel, the saturation channel and the chromaticity channel;

and taking the RGB color image as a target enhancement image corresponding to the target video frame.

In a possible implementation manner of the first aspect, generating an RGB color image according to the enhanced luminance channel, the saturation channel, and the chrominance channel includes:

Dividing the enhanced brightness channel into a plurality of image blocks;

For each image block, carrying out contrast enhancement processing on the brightness of the image block to obtain an enhanced image block corresponding to the image block;

splicing adjacent enhanced image blocks to obtain an updated enhanced brightness channel;

And generating an RGB color image according to the updated enhanced brightness channel, the saturation channel and the chromaticity channel.

In a possible implementation manner of the first aspect, determining, according to the target enhanced image, an enhanced image corresponding to the video frame to be processed includes:

determining a fixed point set and a moving point set of a pixel point of a video frame to be processed relative to a target video frame;

For each stationary point in the stationary point set, taking the pixel value of the pixel point corresponding to the stationary point in the target enhanced image as the pixel value of the stationary point;

determining corresponding pixel points of the motion points in the target enhanced image aiming at each motion point in the motion point set, and determining pixel values of the motion points according to pixel values of a plurality of pixel points in a preset area where the corresponding pixel points are located;

And obtaining an enhanced image of the video frame to be processed according to the pixel values of all the motionless points in the motionless point set and the pixel values of all the motionless points in the motionpoint set.

In a possible implementation manner of the first aspect, determining a set of motionless points and a set of motionless points in pixels of a video frame to be processed with respect to a target video frame includes:

acquiring an optical flow vector of a pixel point aiming at each pixel point in a video frame to be processed, and determining the motion speed of the pixel point according to the optical flow vector of the pixel point;

Determining pixel points with the motion speed greater than a preset value as motion points, and determining a motion point set according to all the motion points;

and determining the pixel points with the motion speed smaller than or equal to a preset value as the motionless points, and determining a motionless point set according to all motionless points.

In a possible implementation manner of the first aspect, determining the pixel value of the motion point according to the pixel values of the plurality of pixel points in the preset area where the corresponding pixel point is located includes:

Determining a plurality of pixel points in a preset area where the corresponding pixel points are located;

Determining an average value of pixel values of a plurality of pixel points;

The average value is taken as the pixel value of the motion point.

In a second aspect, an embodiment of the present application provides a video enhancement apparatus, including:

The acquisition module is used for acquiring a target video frame in the video to be processed;

the first enhancement module is used for carrying out image enhancement processing on the target video frame to obtain a target enhanced image corresponding to the target video frame;

the first determining module is used for determining an enhanced image corresponding to the video frame to be processed according to the target enhanced image; the video frame to be processed is determined according to the target video frame and is subjected to image enhancement processing;

The execution module is used for taking the video frame to be processed as a target video frame and taking the enhanced image corresponding to the video frame to be processed as a target enhanced image, and returning to execute the step of determining the enhanced image corresponding to the video frame to be processed according to the target enhanced image until the enhanced images respectively corresponding to the preset number of video frames in the video to be processed are obtained;

the second determining module is used for determining the enhanced video corresponding to the video to be processed according to the enhanced image corresponding to the video frame in the video to be processed.

In a third aspect, an embodiment of the present application provides a video processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the steps of any one of the methods of the first aspect when the computer program is executed by the processor.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which when executed by a processor performs the steps of any of the methods of the first aspect described above.

In a fifth aspect, an embodiment of the application provides a computer program product for, when run on a terminal device, causing the terminal device to perform the method of any of the first aspects described above.

In the video enhancement method provided by the embodiment of the application, the enhancement image corresponding to each video frame to be processed is obtained based on the target enhancement image corresponding to the target video frame, so that the pixel values of each pixel point on the enhancement image corresponding to each video frame to be processed and each target video frame are ensured to have relevance, and therefore, brightness jitter is not generated between the enhancement images corresponding to each video frame to be processed and each target video frame; each group of adjacent video frames in the enhanced video obtained by the method is an enhanced image corresponding to each group of to-be-processed video frames and target video frames in the to-be-processed video, so that pixel values of all pixel points on the adjacent video frames in the enhanced video have relevance, and brightness jitter is not generated.

It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of a video enhancement method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a target video frame color information model according to an embodiment of the present application;

FIG. 3 is a flowchart of a method for acquiring an enhanced image of a target according to an embodiment of the present application;

FIG. 4 is a schematic diagram of capturing an enhanced image of a target provided by an embodiment of the present application;

FIG. 5 is a flow chart of determining a blended illumination component of a target video frame according to an embodiment of the present application;

FIG. 6 is a flow chart of generating a target enhanced image according to another embodiment of the present application;

FIG. 7 is a flow chart of generating a target enhanced image according to yet another embodiment of the present application;

FIG. 8 is a flowchart illustrating a method for determining an enhanced image corresponding to a video frame to be processed according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of a video enhancement device according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of a video processing apparatus according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

The video enhancement technology has wide application in the computer vision fields of urban traffic, video monitoring, intelligent vehicles and the like. Currently, video enhancement is generally implemented by enhancing individual images of each video frame in a video to implement enhancement processing of the entire video segment, where the image enhancement method of the video frame includes one or more of the following: histogram equalization, chromaticity mapping, retinex theory-based method, wavelet change method, image fusion.

However, the image enhancement algorithms are all used for enhancing each frame of video frame in the video independently, and do not consider the association relationship between multiple frames of video frames in the video, so that the brightness change of the pixel points at the same position between adjacent video frames is easy to be large, and the problems of brightness jitter or flickering and the like occur.

The technical scheme of the present application and how the technical scheme of the present application solves the above technical problems are exemplarily described below with specific embodiments. It is noted that the specific embodiments listed below may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

Fig. 1 is a flow chart of a video enhancement method according to an embodiment of the present application, where an execution body of the embodiment is a video processing device; the video processing device includes, but is not limited to, a mobile terminal such as a smart phone, a tablet computer, a wearable device, and the like, and may also be a desktop computer, a robot, a server, and the like. The video enhancement method as shown in fig. 1 may include:

S10, acquiring a target video frame in the video to be processed.

In this embodiment, the video to be processed may be a video acquired by the video processing device from a mobile terminal such as a video acquisition device, or may be a video acquired by the video processing device from a server, or may be a video stored in advance on the video processing device.

The video acquisition equipment can be a video camera device, a mobile terminal with a video camera function, a tablet personal computer, an intelligent robot and the like.

The server may be a cloud server.

Optionally, the video to be processed may be the acquired complete video, or may be a video segment within a preset time range in the acquired video.

For example, the video processing device acquires 24 hours of monitoring video of 3 months and 20 days sent by the monitoring device, and a video segment with a start time of 18:00 and an end time of 19:00 is a video to be processed.

In this embodiment, the video to be processed includes N consecutive video frames, where N is greater than or equal to 2 and N is an integer; the target video frame can be any frame of video frame in the video to be processed; for example, the target video frame is an r frame in the video to be processed, where r is greater than or equal to 1 and less than or equal to N.

For example, after the video processing device obtains the video to be processed, the slicing time interval of the video to be processed may be determined according to the frame rate of the enhanced video, and the video to be processed may be divided into N continuous video frames according to the time interval, and then one video frame is randomly selected from the N continuous video frames as the target video frame. The frame rate is the number of video frames played by the enhanced video in each second, and the frame rate of the enhanced video is a preset value for a user.

For example, the video to be processed is a video segment with a start time of 17:00 and an end time of 17:10, and if the frame rate of the enhanced video is 24, the slicing time interval is 1/24s; the video to be processed is segmented at 1/24s as time intervals, 14400 (obtained by segmenting 600s of the video to be processed) continuous video frames are obtained, and any frame in the 14400 video frames is selected as a target video frame.

S20, performing image enhancement processing on the target video frame to obtain a target enhanced image corresponding to the target video frame.

In this embodiment, the video processing apparatus may select different image enhancement processing modes according to the image enhancement requirement of the target video frame.

For example, the video to be processed is a video obtained by shooting under a low-illumination condition such as night, and at this time, the pixel values of most of the pixel points of the target video frame are concentrated in a low-gray area, and the image enhancement of the target video frame is required to improve the contrast of the image. The video processing device can perform nonlinear stretching on the gray values of the pixel points in the target video frame, so that the gray of the target video frame is uniformly distributed, and the contrast of the target video frame is improved.

For example, the video frame to be processed is a video obtained by shooting under the condition of uneven illumination, and at this time, the target video frame has an artifact due to uneven illumination, and the image enhancement of the target video frame is required to eliminate the artifact. Since the color information that an object can observe is determined by the reflective power of the object for different light waves (reflective properties) and the intensity of incident light that impinges on the object, the reflective power of the object for different light waves does not change due to the change in the intensity of incident light, and the reflective power of the object for different light waves determines the color of the object itself. Therefore, in order to eliminate the artifacts caused by uneven illumination of the target video frame, the video processing device may remove the influence of the incident light in the target video frame by performing filtering processing on the target video frame, and retain the reflection attribute of the object, thereby implementing image enhancement of the target video frame.

In one example, a model schematic view of color information of a target video frame may be shown in fig. 2, where the target video frame may be understood that the video acquisition device acquires an original image S ', and the color information on the target video frame is determined by the reflectivity of an object in the target video frame to incident light and the intensity of the incident light irradiated on the object, and then the expression of S' may be

S'＝L'×R' (1)

Wherein S ' is a target video frame, L ' is an illumination component representing illumination intensity information of incident light, and R ' is a reflection component representing reflection capability of an object on the incident light.

When the incident light is uneven, artifacts appear on the target video frame, and at this time, the true color information of the object in the target video frame cannot be obtained.

In order to eliminate the artifact on the target video frame, the video processing apparatus first performs smoothing filter processing on the target video frame S ' to restore the illumination component L ' of the target video frame as much as possible, then uses the illumination component and the target video frame as inputs, determines the S ' reflection component L ' of the target video frame by the formula (1), and determines the reflection component L ' as a target enhanced image corresponding to the target video frame.

According to the method, the reflection component of the reflection capability of the reflecting object is obtained by removing the illumination component of the target video frame, so that the artifact generated by uneven illumination (illumination component) can be effectively eliminated, and the enhancement of the object in the target video frame is realized.

In this embodiment, the image enhancement processing method of the target video frame may be one or more of histogram equalization, wavelet transformation, chromaticity mapping, and image fusion imaging, which is not limited herein.

S30, determining an enhanced image corresponding to the video frame to be processed according to the target enhanced image.

In this embodiment, the video frame to be processed is a video frame to be subjected to image enhancement processing, which is determined according to the target video frame.

The video frame to be subjected to the image enhancement processing may be a video frame separated from the target video frame by a preset frame number, or may be a video frame adjacent to the target video frame.

Alternatively, the video frame to be subjected to the image enhancement process may be a video frame separated from the target video frame by a preset number of frames. The value of the preset frame number can be determined by the ratio of the frame rate of the video to be processed to the frame rate of the enhanced video.

For example, assuming that the video to be processed is obtained by dividing the acquired video by the video processing apparatus according to the slicing time interval in step 10, the slicing time interval may be set according to the frame rate of the enhanced video, so that the frame rate of the video to be processed is n times the frame rate of the enhanced video, where n is an integer greater than or equal to 1.

Specifically, the slicing time interval t may be expressed as:

t＝1/nfv (2)

Where fv is the frame rate of the enhanced video. The frame rate of the video to be processed is 1/t, i.e., nfv, such that the frame rate of the processed video is n times the frame rate of the enhanced video.

If the ratio of the frame rate of the video to be processed to the frame rate of the enhanced video is n, the preset frame number is n-1.

For example, if n is 2, it means that an enhanced image is obtained for every 2 video frames in the video to be processed, that is, the video frames to be processed are subjected to image enhancement processing once every other video frame, so the preset frame number is 1.

In this embodiment, the video frame separated from the target video frame by the preset frame number may be a video frame separated from the target video frame by the preset frame number and having a playing time later than that of the target video frame, or may be a video frame separated from the target video frame by the preset frame number and having a playing time earlier than that of the target video frame.

In one example, if the target video frame is the first video frame of the video to be processed (i.e., the first frame of the video to be processed), the video frame to be processed is a video frame separated from the target video frame by a preset number of frames and having a playback time later than the target video frame. For example, the video to be processed includes 256 continuous video frames, and the 256 video frames are numbered sequentially by 1 and 2 … in order according to the playing order; assuming that the target video frame is the 1 st video frame in the video to be processed, and the preset frame number is 1, the video frame to be processed is the 3 rd video frame in the video to be processed.

In another example, if the target video frame is the last video frame of the video to be processed (for example, the video frame with the latest playing time in the video to be processed), the video frame to be processed is a video frame separated from the target video frame by a preset number of frames and having a playing time earlier than that of the target video frame. For example, the video to be processed includes 256 continuous video frames, and the 256 video frames are numbered sequentially by 1 and 2 … in order according to the playing order; assuming that the target video frame is the 256 th video frame in the video to be processed, and the preset frame number is 1, the video frame to be processed is the 254 th video frame in the video to be processed.

In still another example, if the number of video frames between the target video frame and the last video frame is greater than the preset number of frames, and the number of video frames between the target video frame and the first video frame is greater than the preset number of frames, the video frame to be processed may be a video frame separated from the target video frame by the preset number of frames and having a play time later than that of the target video frame, or may be a video frame separated from the target video frame by the preset frame and having a play time earlier than that of the target video frame. For example, the video to be processed includes 256 continuous video frames, and the 256 video frames are numbered sequentially by 1 and 2 … in order according to the playing order; assuming that the target video frame is the 100 th video frame in the video to be processed and the preset frame number is 1, the video frame to be processed is the 102 th video frame or the 98 th video frame in the video to be processed.

In this embodiment, the video frame to be subjected to the image enhancement process may also be a video frame adjacent to the target video frame. A video frame adjacent to a target video frame may be understood as a video frame separated from the target video frame by a number of frames of 0; that is, when the preset frame number is 0, the video frame to be processed is a video frame adjacent to the target video frame.

Based on this, the video frame adjacent to the target video frame may be a video frame adjacent to the target video frame and having a play time later than that of the target video frame, or may be a video frame adjacent to the target video frame and having a play time earlier than that of the target video frame.

In this embodiment, the purpose of determining, by the video processing apparatus, an enhanced image corresponding to a video frame to be processed according to a target enhanced image is to update, according to pixel values of pixel points on the target enhanced image on which image enhancement processing has been performed, pixel values of pixel points on the video frame to be processed on which image enhancement image processing has not been performed, to obtain an enhanced image of the video frame to be processed, so that pixel values of respective pixel points on the enhanced images corresponding to the video frame to be processed and the target video frame have relevance.

For example, if the video to be processed includes N consecutive video frames, where N is greater than or equal to 2 and N is an integer; the target video frame is an r-th frame in the video to be processed, wherein r is more than 2 and less than N-1, and the video frame to be processed is a video frame adjacent to the target video frame; then the enhanced images respectively corresponding to the two video frames (the (r+1) th frame and the (r-1) th frame) adjacent to the target video frame (the (r) th frame) can be obtained simultaneously according to the enhanced images of the target video frame; the method for obtaining the enhanced image corresponding to each of the (r+1) th frame and the (r-1) th frame may be the same. An exemplary description will be given below of determining an enhanced image corresponding to the (to-be-processed) th video frame (to-be-processed video frame) from the enhanced image (target enhanced image) of the (r) th video frame.

Illustratively, determining, from the enhanced image of the r-th video frame, the enhanced image corresponding to the r+1-th video frame may include:

and A1, acquiring a fixed point set and a moving point set of each pixel point of the (r+1) th video frame relative to the (r) th video frame.

Optionally, for each pixel point on the (r+1) th video frame, the brightness change rate of the pixel point relative to the pixel point on the same coordinate position on the (r) th video frame is taken as a speed vector of the pixel point, the pixel point with the speed vector larger than a preset value is determined as a motion point, and the pixel point with the speed vector smaller than or equal to the preset value is determined as a stationary point. Based on the method, each pixel point on the (r+1) th video frame is determined to be an motionless point or a motion point relative to the (r) th video frame, then a motion point set is determined according to all the motion points on the (r+1) th video frame, and a motionless point set is determined according to all the motionless points on the (r+1) th video frame.

In another example, the pixel values of the pixels at the same coordinate position on the (r+1) th video frame and the (r) th video frame are subjected to difference processing, so as to obtain pixel difference values corresponding to the pixels on the (r+1) th video frame. Since the object to be photographed may be stationary or moving in two adjacent video frames during video photographing, the positions and pixel values of the pixels representing the stationary object on the two adjacent video frames are the same, and the positions and pixel values of the pixels representing the contour of the moving object on the two adjacent video frames are different.

Thus, the position and contour of the (r+1) th video frame relative to the plurality of moving objects on the (r) th video frame can be determined from the pixel points whose pixel difference value is not 0.

Then, taking the pixel points on the outline and inside of the moving object as the motion points of the (r+1) th video frame relative to the (r) th video frame, and determining a motion point set according to all the motion points; and taking the pixel points outside the moving object outline as the motionless points of the (r+1) th video frame relative to the (r) th video frame, and determining a motionless point set according to all motionless points.

A2, regarding each fixed point in the fixed point set, taking the pixel value of the pixel point corresponding to the fixed point in the enhanced image of the r-th video frame as the pixel value of the fixed point

The pixel point corresponding to the motionless point in the enhanced image of the (r) th video is the pixel point with the same coordinate as the motionless point in the (r+1) th video frame.

For example, assuming that the coordinates of the stationary point SP1 at the (Xp 1, yp 1) th video frame are (x 1, yp 1), the pixel corresponding to the stationary point SP1 is the pixel with the coordinates of (Xp 1, yp 1) on the enhanced image of the (r) th video frame.

In this step, the pixel value of the pixel corresponding to the stationary point in the enhanced image of the r-th video frame is used as the pixel value of the stationary point, which may mean that the pixel having the same coordinates as the stationary point in the enhanced image of the r-th video frame is determined, and the pixel value of the pixel is used as the pixel value of the stationary point.

For example, assuming that the stationary point SP1 is at coordinates (Xp 1, yp 1) of the (r+1) th video frame, the pixel value of the stationary point SP1 is at the pixel value of the pixel point at coordinates (Xp 1, yp 1) on the enhanced image of the (r) th video frame.

And A3, determining a corresponding pixel point of the motion point in the enhanced image of the r-th video frame aiming at each motion point in the motion point set, and determining the pixel value of the motion point according to the pixel values of two pixel points adjacent to the corresponding pixel point.

The coordinates of the corresponding pixel point of the motion point in the enhanced image of the r-th video frame can be obtained by calculating according to the speed vector of the motion point and the coordinates of the motion point in the r+1th video frame.

For example, assuming that the coordinates of the motion point JP1 on the (x 1, yp 1) th video frame are (v _x,v_y) and the velocity vector is (v _x,v_y), the coordinates of the corresponding pixel point of the motion point JP1 in the enhanced image of the (r) th video frame are (x 1+ v _x×△t,yp1+v_y ×Δt). Where Δt is the time interval between the (r+1) th video frame and the (r) th video frame.

In this step, determining the pixel value of the motion point from the pixel values of the two pixels adjacent to the corresponding pixel point may refer to taking the average value of the pixel values of the two pixels adjacent to the corresponding pixel point as the pixel value of the motion point.

The two pixels adjacent to the corresponding pixel may be two pixels adjacent to the corresponding pixel in the X direction or two pixels adjacent to the corresponding pixel in the Y direction.

For example, assuming that the coordinates of the corresponding pixel point JP1 'of the motion point JP1 in the enhanced image of the r-th video frame are (xp 1', yp1 '), the resolution of the enhanced image of the r-th video frame in the X direction is i, and the resolution in the Y direction is j, the coordinates of two pixel points adjacent to JP1' may be (xp 1'+i, yp 1) and (xp 1' -i, yp 1), or may also be (xp 1', yp1' +j) and (xp 1', yp1' -j).

After coordinates of two pixel points adjacent to JP1' are obtained, an average value of pixel values of the two pixel points is taken as a pixel value of the motion point JP 1.

And step A4, the video processing equipment combines all the fixed points in the fixed point set and all the moving points in the moving point set to obtain an enhanced image of the (r+1) th video frame.

The purpose in this step is to obtain an enhanced image of the (r+1) th video frame by combining the coordinates of each pixel point (including the stationary point and the moving point) after obtaining the pixel value of each stationary point in the stationary point set and the pixel value of each moving point in the moving point set.

Illustratively, the set of stationary points includes 6 stationary points, and the pixel values of the 6 stationary points are represented as Sp1 (1, 1), sp2 (1, 2), sp3 (1, 3), sp4 (2, 1), sp5 (3, 1), and Sp6 (3, 2), respectively; the motion point set comprises 3 motion points, and the pixel values of the 3 motion points are Jp1 (2, 2), jp2 (2, 3) and Jp3 (3, 3) respectively; wherein, (x, y) is the coordinates of each pixel point.

The 9 pixel points are combined according to the coordinate positions, and the enhanced image of the (r+1) th video frame can be obtained by referring to the following table 1:

TABLE 1 Pixel composition Table for enhanced images

Sp1(1,1)	Sp2(1,2)	Sp3(1,3)
			Sp4(2,1)	Jp1(2,2)	Jp2(2,3)
Sp5(3,1	Sp6(3,2)	Jp3(3,3)

S40, taking the video frames to be processed as target video frames, taking the enhanced images corresponding to the video frames to be processed as target enhanced images, and returning to execute the step of determining the enhanced images corresponding to the video frames to be processed according to the target enhanced images until the enhanced images respectively corresponding to the preset number of video frames in the video to be processed are obtained.

In this embodiment, in the step of returning to execute the enhanced image corresponding to the video frame to be processed according to the target enhanced image, the video frame to be processed is a video frame which is separated from the target enhanced video by a preset number of frames and has not been subjected to image enhancement processing.

For example, the video to be processed includes 256 video frames, and the target video frame obtained in step 30 for the first time is the 100 th video frame, and the preset frame number is 1.

In the first cycle, the target video frame is the 100 th video frame, and the video frame to be processed is the 102 th video frame or the 98 th video frame.

If the video frame to be processed is the 102 th video frame; then in the second cycle, the target video frame is the 102 th video frame, and the video frame to be processed may only be the 104 th video frame (the video frame separated from the 102 th video frame by 1 frame and not subjected to the image enhancement processing). In the subsequent cycle, the target video frames are all video frames which are separated from the target enhanced video by 1 video frame and have a playing time later than that of the target video frames.

If the video frame to be processed is the 98 th video frame; in the second cycle, the target video frame is the 98 th video frame, and the video frame to be processed can only be the 96 th video frame (the video frame which is separated from the 98 th video frame by 1 frame and is not subjected to image enhancement processing yet); in the subsequent cycle, the target video frames are all video frames which are separated from the target enhanced video by 1 video frame and have a playing time earlier than that of the target video frames.

If in the first cycle, if the video processing device acquires the enhanced images of the 102 th video frame and the 98 th video frame through two threads at the same time, for example, thread 1 is used for acquiring the enhanced image of the 102 th video frame, and the user of thread 2 acquires the target video frame as the 98 th video frame; in the second cycle, the target video frame of the thread 1 is the 102 th video frame, and the video frame to be processed is the 104 th video frame; the target video frame of the thread 2 is the 98 th video frame, and the video frame to be processed is the 96 th video frame. In the subsequent cycle, the target video frames in the thread 1 are all video frames which are separated from the target enhanced video by 1 video frame and have a playing time later than that of the target video frames; the target video frames in thread 2 are all video frames separated from the target enhanced video by 1 video frame, and the playing time is earlier than that of the target video frames.

In this embodiment, the preset number may be determined by the number of frames of the video to be processed and the preset number of frames.

For example, the video to be processed includes N video frames, N is greater than or equal to 2 and N is an integer. Assuming that the preset frame number is 1, namely determining a video frame to be processed every 1 frame; if N can be divided by 2, then enhanced images of N/2 video frames can be obtained; if N is not divisible by 2, enhanced images of N-1/2 video frames to be processed can be obtained.

In this embodiment, the video processing apparatus obtains enhanced images corresponding to a preset number of video frames in the video to be processed, respectively.

For example, the video to be processed includes 256 video frames, the numbers of the 256 video frames are sequentially 1,2,3 … and 256, and the preset number is 128 if the preset number is 1. Assuming that the target video frame obtained by the video processing apparatus for the first time is the 100 th video frame, in the first cycle, the video processing apparatus obtains an enhanced image of the 102 th video frame through thread 1, and thread 2 obtains an enhanced image of the 98 th video frame.

Thread 1 acquires an enhanced image of the 104 th video frame in the second cycle; acquiring an enhanced image of the 106 th video frame in a third cycle; by analogy, thread 1 sequentially obtains enhancement videos corresponding to video frames numbered 102, 104 … 256 in multiple loops. I.e. thread 1 obtains a total of 78 video frames of each respective corresponding enhanced image.

Thread 2 obtains an enhanced image of the 98 th video frame in the second cycle; acquiring an enhanced image of the 96 th video frame in a third cycle; by analogy, thread 2 sequentially obtains enhancement videos corresponding to video frames numbered 98, 96, … 2 in multiple cycles. I.e. thread 2 obtains a total of 49 video frames each respectively corresponding enhanced images.

At this time, the video processing apparatus obtains a total of 128 (preset number) video frames of 1+78+49 respectively corresponding enhanced images.

S50, determining the enhanced video corresponding to the video to be processed according to the enhanced image corresponding to the video frame in the video to be processed.

The step 40 of obtaining the enhanced images corresponding to the video frames in the video to be processed refers to obtaining the enhanced images corresponding to the preset number of video frames respectively.

In this embodiment, after obtaining enhancement images corresponding to each of the preset number of video frames, the video processing apparatus sorts the preset number of video frames according to the identifiers of the video frames to obtain a video segment composed of the preset number of video frames, and then replaces each video frame in the video segment with a corresponding enhancement image to obtain an enhancement video.

The video frame identifier may be a play time, a sequence number, etc.

For example, the video to be processed includes 32 consecutive video frames, and the sequence numbers of the 32 video frames are sequentially 1 and 2 …. The sequence numbers are used for representing the playing sequence of N video frames in the video to be processed, and the smaller the sequence numbers are, the earlier the playing time is.

Assuming that the video frame to be processed is a video frame separated from the target video frame by one frame, the target video frame obtained by the video processing device for the first time is the 16 th video frame, and in the first cycle, the video processing device obtains an enhanced image of the 18 th video frame through the thread 1, and the thread 2 obtains an enhanced image of the 14 th video frame. After multiple iterations, thread 1 obtains enhanced images corresponding to video frames with sequence numbers of 18, 20, 22 … and 32 respectively, and thread 2 obtains enhanced images corresponding to video frames with sequence numbers of 14, 12, 10 and … 2 respectively, that is, video processing equipment obtains enhanced images corresponding to 16 video frames respectively, and the sequence numbers of the 16 video frames are 2,4, 6, … and 32 respectively; then sequencing the 16 video frames according to sequence numbers to obtain an intermediate video; and finally, replacing each video frame in the intermediate video with a corresponding enhanced image to obtain an enhanced video.

In the video enhancement method provided by the embodiment of the application, the enhancement image corresponding to each video frame to be processed is obtained based on the target enhancement image of the target video frame which is separated from the video frame to be processed by a preset frame number, so that the pixel values of each pixel point on the enhancement images corresponding to the video frame to be processed and the target video frame are ensured to have relevance, and therefore, brightness jitter is not generated between the enhancement images corresponding to the video frame to be processed and the target video frame; because each group of adjacent video frames in the enhanced video corresponds to an enhanced image corresponding to each group of to-be-processed video frames and target video frames in the to-be-processed video, pixel values of all pixel points on the adjacent video frames in the enhanced video are associated, and brightness jitter is not generated.

Fig. 3 is a schematic flow chart of acquiring a target enhanced image according to an embodiment of the present application, which describes one possible implementation manner of acquiring a target enhanced image corresponding to a target video frame in step 20 in the embodiment of fig. 1. As shown in fig. 3, performing image enhancement processing on a target video frame to obtain a target enhanced image corresponding to the target video frame, including:

S201, acquiring a brightness channel of a target video frame.

The video frame in the video to be processed acquired by the video acquisition equipment is usually an RGB (Red, green, blue) color image, which comprises three color channels of red R, green G and blue B, the three color channels all comprise brightness information, the brightness of the three color channels is different, and if the three channels are respectively subjected to image enhancement processing, the proportion of R, G, B is easily changed, so that the color distortion of the acquired enhanced image is caused. In order to avoid color distortion of the enhanced image, enhancement processing is generally performed only on the luminance channel of the target video frame, so that the luminance channel of the target video frame needs to be acquired first.

In this embodiment, the video processing apparatus obtains the luminance channel of the target video frame, which may include the following steps:

And D1, converting the target video frame from the RGB color space to a color space containing a brightness channel, and obtaining a converted target video frame.

In this step, the color space including the luminance channel may be an HSV color space, a YCbCr color space, or the like, which is not particularly limited herein.

In the HSV color space, H is Hue and represents chromaticity; s is Saturation, V is Value, and brightness. In the YCbCr color space, Y is a luminance component, cb is a blue chrominance component, and Cr is a red chrominance component.

In this step, the conversion of the target video frame from the RGB color space to the color space containing the luminance channel (may be referred to as a target color space) may refer to obtaining parameter values corresponding to each pixel point in the target video frame in the target color space.

Illustratively, the target video frame includes a plurality of pixels, and the pixel value of each pixel in the RGB color space may be represented as (R, G, B), wherein R, G, B is a value between 0 and 255. For example, the pixel value of the pixel f is (0, 100, 255).

Assuming that the target color space is the HSV color space, the parameter values (H, S, V) corresponding to each pixel point of the target video frame in the HSV color space may be sequentially determined.

Wherein the pixel value of the pixel in the RGB color space is (R, G, B), the chromaticity value H of the pixel can be shown in the following formula (3)

Wherein, R ', G', B 'are reference values obtained by respectively normalizing the R value, G value, and B value of the pixel point, and R' =r/255; g' =g/255; b' =b/255;

cmax is the maximum value of the reference values, cmax=max (R ', G ', B ');

cmin is the minimum of the reference values, cmin=min (R ', G ', B ');

Δ＝Cmax-Cmin。

the saturation value S of the pixel point can be referred to as the following formula (4)

Wherein Cmax, delta are the same as those explained in formula (3).

The brightness V of the pixel f can be expressed by the following formula (5)

V＝Cmax (5)

Wherein Cmax is the same as explained in formula (3).

And (3) obtaining the H value, the S value and the V value corresponding to each pixel point of the target video frame in the HSV color space based on formulas (3), (4) and (5), thereby realizing the conversion of the target video frame from the RGB color space to the HSV color space.

And D2, extracting and obtaining a brightness channel of the target video frame according to the converted target video frame.

In this step, the luminance channel of the converted target video frame may be used as the luminance channel of the target video frame.

For example, assuming that the converted target video frame is an HSV color space, the converted target video frame includes three channels of chromaticity H, saturation S and luminance V, and the luminance channel V is used as a luminance channel of the target video frame.

After obtaining the luminance channel of the target video frame, the video processing device may perform image enhancement processing on the luminance channel according to the image enhancement requirement of the target video frame to obtain an enhanced luminance channel. And then converting to RGB color channels according to the enhanced brightness channel, the initial chromaticity channel H and the initial saturation S of the target video frame to obtain an enhanced image of the target video frame.

The method for performing image enhancement processing on the brightness channel of the target video frame can be a histogram equalization method, a wavelet change-based method and the like. The video processing device may select different image enhancement processing modes according to the image enhancement requirements of the target video frame. An exemplary manner of image enhancement to remove artifacts is described below by steps 202 through 204.

S202, determining a plurality of illumination components of the target video frame according to the brightness channel of the target video frame.

As can be seen from fig. 2, the core idea of removing the artifacts in the target video frame is to estimate the illumination component of the luminance channel as accurately as possible, and perform enhancement processing on the luminance channel according to the illumination component obtained by the estimation.

The estimation method of the target illumination component is usually obtained by smoothing and filtering the original image, and the illumination component has more image details but lower brightness. For this purpose, the luminance channel may be processed in a plurality of ways to obtain a plurality of illumination components of the luminance channel; and then fusion processing is carried out on the plurality of illumination components, so that the obtained fusion illumination components have higher brightness and more image details. The purpose of this step is to obtain a plurality of illumination components of the luminance channel.

In a possible embodiment, the luminance channel is subjected to a filtering process to obtain the first illumination component L1 of the luminance channel.

For example, a luminance channel is subjected to a guided filter process, or a gaussian filter process.

Illustratively, the gaussian function G (x, y) may be as shown in equation (6):

Wherein x and y represent coordinate values of pixel points on a brightness channel, lambda is a constant, and sigma is a scale parameter.

When sigma is small, the obtained illumination component can keep good edge detail; the value of σ may be set in advance by the user. Alternatively, the scale parameter σ may be set to a smaller value, so that the first illumination component L1 may maintain better edge detail.

In another possible embodiment, in order to make the estimated illumination component have a higher luminance, a variation operation may be performed on the luminance channel to obtain a second illumination component L2 of the luminance channel. The second illumination component estimated via the variance method has a higher brightness, but may lose some detailed information.

In the variation operation, the illumination component estimation of the brightness channel is converted into an optimal solution problem of quadratic programming, and the second illumination component L2 is estimated by a gradient descent method after an objective function is set.

Optionally, the objective function of the variation operation is as shown in equation (7):

wherein, alpha, beta and gamma are weight coefficients, D is a difference operator, L is an illumination component of the brightness channel V, and R is a reflection component of the brightness channel V.

S203, determining a fusion illumination component corresponding to the target video frame according to the illumination components of the target video frame.

The first illumination component L1 obtained by the step 202 has edge detail information which can be better, and the second illumination component L2 obtained has higher brightness, and the purpose of the step is to perform fusion processing on the first illumination component L1 and the second illumination component L2 to obtain a fused illumination component, so that the fused illumination component has better detail information and higher brightness.

It should be appreciated that if more than two illumination components are obtained in step 202, then fusion processing may be performed on the more than two illumination components to obtain a fused illumination component for the target video frame.

In this embodiment, determining the fusion lighting component corresponding to the target video frame according to the multiple lighting components of the target video frame may refer to respectively obtaining gradients corresponding to the first lighting component L1 and the second lighting component L2, and performing fusion processing on the first lighting component L1 and the second lighting component L2 by using the gradients as weight values to obtain the fusion lighting component corresponding to the target video frame.

The gradient of the first illumination component L1 may refer to an average gradient of the first illumination component L1, or may be a gradient of each pixel point on the first illumination component L1. The gradient of the second illumination component L2 may refer to an average gradient of the second illumination component L2, or may be a gradient of each pixel point on the second illumination component L2.

Illustratively, the gradients of L1 and L2 are each the gradient of each pixel point.

Taking a pixel point O as an example, wherein the gradient of the pixel point O on L1 is J, and the pixel value is f ₁; the gradient of the pixel point O on L2 is K, and the pixel value is f2, and the pixel value f ₃ after the pixel point is fused is:

f₃＝f₁×J+f₂×K (8)

And (3) processing each pixel point according to the formula (8) to obtain a fused pixel value of each pixel point. And then changing the pixel value of each pixel point on the first illumination component or the second illumination component into a fused pixel value to obtain a fused illumination component corresponding to the target video frame.

S204, processing the target video frame according to the fusion illumination component to generate a target enhanced image corresponding to the target video frame.

After the fusion illumination component is obtained, determining a reflection component of the target video frame based on the brightness channel of the fusion illumination component and the target video frame, and taking the reflection component as an enhanced brightness channel of the target video frame; it should be noted that, because the fused illumination component has both better detailed information and higher brightness, the reflection component determined based on the fused illumination component can accurately embody the characteristics of the object in the video frame, thereby realizing the enhancement of the brightness channel of the target video frame.

Since the color space where the target video frame and the target enhanced image are located needs to be kept unchanged, after the enhanced luminance channel of the target video frame is obtained, the target enhanced image which is in the same color space as the target video frame needs to be obtained by conversion based on the enhanced luminance channel and other channels in the color space where the enhanced luminance channel is located.

For example, the target video frame is an image in RGB color space, the color space in which the enhanced luminance channel is located is HSV color space, and the enhanced luminance channel is a luminance channel V in HSV color space, so the video processing apparatus needs to further obtain a chrominance channel H of the target video frame and a saturation channel S of the target video frame, and convert the chrominance channel H and the saturation channel S into a color image in corresponding RGB color space according to the luminance channel V, the chrominance channel H and the saturation channel S, and use the color image as a target enhanced image corresponding to the target video frame.

For a clearer description of the present embodiment, please refer to fig. 4, fig. 4 is a schematic diagram of generating an enhanced image of a target. As shown in fig. 4, the target video frame is a color image of an RGB color space, and the target video frame may be denoted as f (R, G, B), and the target video frame is first converted from the RGB color space to the HSV color space to obtain a chrominance channel H, a saturation channel S and a luminance channel V; the enhancement processing is performed on the brightness channel V to obtain an enhanced brightness channel V ', then the color space conversion is performed on the chrominance channel H, the saturation channel S and the brightness channel V ', the color space conversion is performed on the chrominance channel H, the saturation channel S and the brightness channel V ' to obtain R1, G1 and B1, wherein R1 represents a red channel, G1 represents a green channel and B1 represents a blue channel, and the R1, G1 and B1 are combined to generate a target enhanced image f (R1, G1 and B1).

The process of converting the color space of the chrominance channel H, the saturation channel S and the luminance channel V' to the RGB color space to obtain R1, G1, B1 may be determined based on formulas (3), (4), (5).

In the application scene applied to removing the artifact of the target video frame in the embodiment, the brightness channel is processed in multiple modes (two modes), the multiple illumination components are estimated, and the multiple illumination components are fused to obtain the fused illumination component, so that the fused illumination component has better detail information and higher brightness, and therefore, the reflection component determined based on the fused illumination component can accurately reflect the characteristics of the object, and the enhancement of the target video frame is realized.

Fig. 5 is a schematic flow chart of determining a fusion illumination component of a target video frame according to an embodiment of the present application, which describes a possible implementation of the fusion processing of two illumination components in step S203 in the embodiment of fig. 3. As shown in fig. 5, determining a fused illumination component corresponding to the target video frame according to the plurality of illumination components of the target video frame includes:

s2031, determining a first gradient of the first illumination component and a second gradient of the second illumination component.

The first illumination component L1 may be obtained by performing smoothing filtering processing on the luminance channel. The second illumination component L2 may be obtained by performing a variation operation on the luminance channel.

In this embodiment, the gradient of the first illumination component L1 is an average gradient of all pixel points in the first illumination component; the gradient of the second illumination component is the average gradient of all pixels in the second illumination component.

For example, the formula for calculating the average gradient of the first illumination component L1 may be found in formula (9):

Wherein, For the average gradient of the first illumination component, M and N are the width and height of the first illumination component, and F (i, j) is the luminance value of the pixel (i, j) in the first illumination component.

The step of calculating the average gradient of the second illumination component is the same as the step of calculating the average gradient of the first illumination component, and can be seen in formula (10):

Wherein, For the average gradient of the second illumination component, m and n are the width and height of the second illumination component, and f (i, j) is the luminance value of the pixel (i, j) in the second illumination component.

S2032, based on the first gradient and the second gradient, performing fusion processing on the first illumination component and the second illumination component to obtain a fusion illumination component.

In this embodiment, the video processing device may perform fusion processing on the first illumination component and the second illumination component based on a weighted average method, so that the fusion illumination component may provide more target information, so as to implement enhancement processing on the luminance channel according to the fusion illumination component, and further implement enhancement processing on the target video frame according to the enhanced luminance channel.

Illustratively, the video processing device first determines the weight of the first illumination component and the weight of the second illumination component from the first gradient and the second gradient.

Wherein the weight of the first illumination component and the weight of the second illumination component are determined from the first gradient and the second gradient, see equations (11) and (12), respectively:

Wherein G _GF denotes the weight of the first illumination component, Representing an average gradient (first gradient) of the first illumination component; g _VF denotes the weight of the second illumination component,Representing the average gradient of the second illumination component (second gradient).

After obtaining the weight of the first illumination component and the weight of the second illumination component, the video processing device performs weighted summation on the first illumination component and the second illumination component according to the weight of the first illumination component and the weight of the second illumination component, so as to obtain a fusion illumination component.

Wherein the first illumination component and the second illumination component are weighted and summed according to the weight of the first illumination component and the weight of the second illumination component, and the manner of obtaining the fused illumination component can be seen in formula (13):

L₃＝L₁×G_GF+L₂×G_VF (13)

Where L ₁ represents the first illumination component, L ₂ represents the second illumination component, L ₃ represents the fused illumination component G _GF represents the weight of the first illumination component, and G _VF represents the weight of the second illumination component.

When the video to be processed is a video obtained by shooting under low illumination conditions such as night, the video to be processed has artifacts due to the condition that illumination is uneven, and meanwhile, each video frame in the video is dark due to low illumination, which is unfavorable for later processing, if an enhanced image of each video frame is obtained only based on a mode of removing illumination components, although the problem of the artifacts can be better solved, the processed image can be further darkened, based on the enhanced image, after the reflection components are obtained, correction processing (eliminating the influence of illumination unevenness) can be performed on the illumination components, corrected illumination components are obtained, and a final enhanced image is determined according to the corrected illumination components and the reflection components obtained in the embodiment of fig. 3. The enhanced image is subjected to illumination compensation through the corrected illumination component, so that the problems of artifacts and lower brightness of a video shot under a low illumination condition can be solved. An exemplary description is provided below with respect to the embodiment shown in fig. 6.

Fig. 6 is a flowchart of another embodiment of the present application for generating a target enhanced image, which describes a possible implementation of step S204 in the embodiment of fig. 3. As shown in fig. 6, processing the target video frame according to the fused illumination component, generating a target enhanced image corresponding to the target video frame, including:

s2041, determining a reflection component of the brightness channel according to the fused illumination component and the brightness channel.

The fused illumination component of this step may be the fused illumination component L ₃ in the embodiment shown in fig. 5.

The luminance channel of this step is the luminance channel of step 201 in the embodiment of fig. 3.

And taking the brightness channel as an original image S, fusing the illumination component into the illumination component, and determining the reflection component of the brightness channel based on the formula (1).

S2042, the reflection component is corrected, and a corrected reflection component is generated.

When the video to be processed is a video obtained by shooting under low-illumination conditions such as night, the target video frame often has underexposure, at this time, the contrast of the image is low, and the contrast of the reflection component of the target video frame is low.

The method of correction processing may be, for example, gamma correction.

And S2043, performing nonlinear stretching on the fusion illumination component to generate a stretched fusion illumination component.

The aim of this step is to stretch the fused illumination component non-linearly. Optionally, gradation of the fused illumination component can be graded through two preset gradation thresholds, so as to obtain three sections of gradation of low gradation, middle gradation and high gradation; and then respectively carrying out gray stretching on the three gray levels, and setting different stretching parameters (namely nonlinear stretching) so as to effectively enhance the brightness in a low gray level (namely low brightness) area and reduce the influence of uneven illumination.

The preset two gray thresholds are a first gray threshold and a second gray threshold respectively, wherein the second gray threshold is larger than the first gray threshold and smaller than the maximum gray value of the fusion illumination component, and the first gray threshold is larger than the minimum gray value of the fusion illumination component; the gray level in the fusion light component that is less than the first gray level threshold is taken as a low gray level, the gray level in the fusion light component that is less than the second gray level threshold is taken as a medium gray level, and the gray level in the fusion light component that is greater than or equal to the second gray level threshold is taken as a high gray level. Alternatively, the gray scale of the fused illumination component may be trisected by setting a first gray scale threshold and a second gray scale threshold.

In this step, setting different stretching parameters for different gray levels can be achieved by an arctan function.

For an example, an expression fusing the nonlinear stretching of the illumination component can be found in equation (14):

Where a and b are constants, L ₃ represents the fused illumination component, and L _final represents the fused illumination component after stretching. The arctan function curve is a nonlinear curve whose function is to stretch a region of low pixel value (low gray level) to effectively enhance the luminance in the low luminance region.

S2044, obtaining an enhanced brightness channel according to the stretched fusion illumination component and the corrected reflection component.

After the stretched fusion illumination component and the corrected reflection component are obtained, a brightness channel can be determined by utilizing the stretched fusion illumination component and the corrected reflection component, and the problems of artifacts and underexposure in the image in the video shot and obtained under the low illumination condition can be solved simultaneously because the stretched fusion illumination component reduces the influence of illumination non-uniformity and the corrected reflection component enhances the image contrast; for convenience of description, in the present embodiment, the luminance channel may be referred to as an enhanced luminance channel. For example, in one implementation, the manner of obtaining the enhanced luminance channel according to the stretched fused illumination component and the corrected reflection component may be to perform the integration processing on the stretched fused illumination component and the corrected reflection component according to formula (1) to obtain an adjusted luminance channel, and use the adjusted luminance channel as the enhanced luminance channel, which may, of course, also be determined according to the stretched fused illumination component and the corrected reflection component in other manners.

S2045, processing the target video frame according to the enhanced brightness channel, and generating a target enhanced image corresponding to the target video frame.

Since the color space where the target video frame and the target enhanced image are located needs to remain unchanged, after the enhanced luminance channel of the target video frame is obtained, the target enhanced image in the same color space as the target video frame needs to be obtained by conversion based on the enhanced luminance channel and other channels in the color space where the enhanced luminance channel is located.

For example, the video frames in the video to be processed acquired by the video acquisition device are RGB (Red, green, blue) color images, i.e., the target video frames are typically RGB color images, so that the target enhanced image is also corresponding to the RGB color images.

Optionally, if the color space in which the enhanced luminance channel is located is an HSV color space, the enhanced luminance channel is a luminance channel V in the HSV color space, then the S saturation channel and the chrominance channel H of the target video frame need to be further acquired, and then an RGB color image is generated by conversion according to the enhanced luminance channel, the saturation channel and the chrominance channel, and the RGB color image is used as the target enhanced image.

For example, if the target enhanced image is an RGB color space and the color space in which the enhanced brightness channel is located is an HSV color space, the generating the target enhanced image corresponding to the target video frame may include the steps of:

and E1, acquiring a saturation channel and a chromaticity channel of the target video frame.

The step of obtaining the saturation channel and the chroma channel of the target video frame can be specifically referred to the related description of step S204 in the embodiment of fig. 3, which is not repeated here.

And E2, generating an RGB color image according to the enhanced brightness channel, the saturation channel and the chromaticity channel.

In this step, the generation of the RGB color image according to the enhanced luminance channel, the saturation channel, and the chrominance channel may refer to converting each pixel point from the HSV color space to the RGB color space, where the R value, the G value, and the B value on each pixel point are all values between 0 and 255, and the R value, the G value, and the B value on each pixel point are expressed by the pixel value of the pixel point.

Wherein the process of converting each pixel point from the HSV color space to the RGB color space may be determined based on formulas (3), (4), (5).

And E3, taking the RGB color image as a target enhancement image corresponding to the target video frame.

In practical applications, before converting the enhanced luminance channel, the saturation channel, and the chrominance channel to generate the RGB color image in step 2045, the enhanced luminance channel may be first preprocessed, so that the enhanced luminance channel can more clearly represent the real luminance of the target video frame. An exemplary description is provided below with respect to the embodiment shown in fig. 7.

Fig. 7 is a flowchart of a process for generating a target enhanced image according to another embodiment of the present application, which illustrates a possible implementation of generating an RGB color image according to the enhanced luminance channel, the saturation channel, and the chrominance channel in step S2045 in the embodiment of fig. 6. As shown in fig. 7, generating an RGB color image from the enhanced luminance channel, the saturation channel, and the chrominance channel may refer to:

s2061, dividing the enhanced luminance channel into a plurality of image blocks.

In this step, the video processing apparatus may equally divide the luminance channel into a plurality of image blocks according to the image size of the enhanced luminance channel.

Illustratively, the enhanced luminance channel is an image of size 64×64, and the luminance channel is divided into 64 8×8 image blocks.

S2062, for each image block, performing contrast enhancement processing on the brightness of the image block, to obtain an enhanced image block corresponding to the image block.

The goal in this step is to enhance the contrast of each image block.

The performing contrast enhancement processing on the brightness of the image block may mean that, for each image block, the video processing device calculates to obtain a gray distribution histogram of the image block, and determines a mapping function corresponding to the image block based on the gray distribution histogram; and carrying out gray level transformation on the image block based on the mapping function to obtain an enhanced image block corresponding to the image block so as to achieve the purpose of improving the contrast of the image block.

Alternatively, in order to avoid an excessive increase in contrast per image block, it is necessary to limit the contrast of the enhanced image block corresponding to each image block.

For example, after the gray distribution histogram of each image block is obtained through statistics, the gray distribution histogram obtained through statistics in each image block is cut according to a preset upper limit value, the cut gray values are uniformly distributed on the whole gray interval of the image block to obtain an updated gray distribution histogram, and then a corresponding mapping function is determined according to the updated gray distribution histogram.

And S2063, splicing adjacent enhanced image blocks to obtain an updated enhanced brightness channel.

In this embodiment, the local contrast of the enhanced luminance channel may be improved by performing the blocking processing on the enhanced luminance channel, but if the pixel point of each image block is only transformed by the mapping function in the image block, the processing result of the enhanced luminance channel may be caused to be a block effect (e.g. a luminance abrupt change), which is the purpose of this step is to avoid the block effect of the processed enhanced luminance channel.

Alternatively, adjacent enhanced image blocks may be stitched based on a difference operation to obtain updated enhanced luminance channels.

For example, for a pixel point J on an image block, four image blocks adjacent to the image block to which the pixel point J belongs in the left-right direction and the upper-lower direction may be obtained first, mapping functions corresponding to the four image blocks respectively are determined, gray values of the pixel point are transformed based on the four mapping functions to obtain four mapping values respectively, and then bilinear interpolation is performed on the four values to obtain the updated gray values of the pixel point J. Repeating the above process until updated gray values of all pixel points on the image block are obtained.

It should be understood that, for a pixel of an image block located at the boundary of the enhanced luminance channel, the updated gray value of the pixel may be obtained by stitching only two or three image blocks adjacent to the image block where the pixel is located. For example, for a corner of the enhanced luminance channel, the mapping function of two adjacent image blocks of the image block where the corner is located may be directly used for transformation. The corner point may be any one of an upper left corner, a lower left corner, an upper right corner, and a lower right corner.

S2064, generating an RGB color image according to the updated enhanced luminance channel, saturation channel, and chrominance channel.

In this step, the updated enhanced luminance channel may be used as the enhanced luminance channel of step 2045 in the embodiment of fig. 6, and then an RGB color image may be obtained based on the implementation of step 2045 in the embodiment of fig. 6.

Fig. 8 is a schematic flow chart of determining an enhanced image corresponding to a video frame to be processed according to an embodiment of the present application, which describes a possible implementation manner of determining an enhanced image corresponding to a video frame to be processed according to a target enhanced image in step 30 in the embodiment of fig. 1. As shown in fig. 8, determining an enhanced image corresponding to a video frame to be processed according to a target enhanced image includes:

s301, determining a fixed point set and a moving point set of pixel points of a video frame to be processed relative to a target video frame.

The aim of this step is to determine a set of stationary points and a set of moving points in the pixels of the video frame to be processed relative to the target video frame (between adjacent video frames), so as to subsequently apply different pixel value assignment methods to the pixels in the set of stationary points and the set of moving points, respectively.

Optionally, determining the stationary point set and the moving point set of the pixel points of the video frame to be processed relative to the target video frame may refer to determining, for each pixel point on the video frame to be processed, a brightness change rate of the pixel point relative to the pixel point at the same coordinate position on the target video frame as a speed vector of the pixel point, determining, as the moving point, the pixel point whose speed vector is greater than a preset value, and determining the moving point set according to all the moving points; and determining the pixel points with the speed vector smaller than or equal to a preset value as the motionless points, and determining a motion point set according to all motionless points.

The velocity vector of the pixel point may include a velocity vector u in an X direction and a velocity vector v in a Y direction on an image plane where the video frame to be processed is located, where the X direction and the Y direction are directions of coordinate axes of the image plane where the video frame to be processed is located.

Alternatively, the velocity vector of the pixel point may be expressed as (u, v), and the comparison between the velocity vector and the preset value may refer to the comparison between the vector value of the velocity vector and the preset value. Wherein the vector value of the velocity vector can be characterized as the motion velocity of the pixel.

Illustratively, determining the set of stationary points and the set of moving points in the pixels of the video frame to be processed relative to the target video frame may include the steps of:

Step 1, for each pixel point in a video frame to be processed, acquiring a speed vector of the pixel point, and determining the motion speed of the pixel point according to the speed vector of the pixel point.

The speed vector comprises a speed vector u in the X direction and a speed vector v in the Y direction on the image plane of the video frame to be processed, wherein the X direction and the Y direction are directions of coordinate axes of the pixel points.

For example, assuming that the target video frame is a first video frame of the video to be processed and the video frame to be processed is a second video frame of the video to be processed, the obtaining a velocity vector of each pixel point in the second frame image relative to the first frame image may be:

for each pixel point, respectively acquiring pixel values of the pixel point on a second video frame and a first video frame, performing space-time differentiation processing on a difference value of the two pixel values to obtain a change rate of the pixel value in the X direction and a change rate of the pixel value in the Y direction, taking the change rate of the pixel value in the X direction as a speed vector u, and taking the change rate of the pixel value in the Y direction as a speed vector v.

After obtaining the velocity vector u of each pixel in the X direction and the velocity vector v of each pixel in the Y direction, the movement velocity of each pixel can be found by the formula (15):

Wherein S _(x,y) is the motion speed of the pixel point, u is the velocity vector of the pixel point in the X direction, and v is the velocity vector of the pixel point in the Y direction. Wherein, (x, y) are coordinates of the pixel point.

And step 2, determining the pixel points with the motion speed larger than a preset value as motion points, and generating a motion point set according to all the motion points.

And step 3, determining the pixel points with the motion speed smaller than or equal to a preset value as the motionless points, and generating a motionless point set according to all motionless points.

The preset value may be preset by a user or may be preset by a system, and the motionless point set FP formed by all motionless points is obtained after the steps 2 and 3, and the motional point set DP formed by all motional points is obtained.

S302, regarding each fixed point in the fixed point set, taking the pixel value of the pixel point corresponding to the fixed point in the target enhanced image as the pixel value of the fixed point.

The pixel points corresponding to the motionless points in the target enhanced image refer to the pixel points with the same coordinates as the motionless points in the video frame to be processed.

For example, assuming that the coordinates of the motionless point FP1 in the video frame to be processed are (Fx 1, fy 1), the pixel corresponding to the motionless point FP1 is the pixel with the coordinates of (Fx 1, fy 1) on the target enhanced image.

In this step, the pixel value of the pixel corresponding to the stationary point in the enhanced image of the r-th video frame is used as the pixel value of the stationary point, which may mean that the pixel having the same coordinates as the stationary point in the enhanced image of the r-th video frame is determined, and the pixel value of the pixel is used as the pixel value of the stationary point.

For example, the target enhanced image corresponding to the target video frame is I _t, and the video frame to be processed is I _t+1, the pixel value of the stationary point P on the video frame to be processed I _t+1 may be referred to as formula (16):

Wherein, For the pixel value of the motionless point P in the enhanced image of the video frame I _t+1 to be processed,Is the pixel value of the same pixel point as the P position in the target enhanced image I _t.

Illustratively, the target video frame is a first video frame of the video to be processed, and the video frame to be processed is a second video frame of the video to be processed. The P point is the fixed point of the second video frame relative to the first video frame, and the pixel value of the P point in the enhanced image of the second video frame is the same as the pixel value of the same pixel point as the P point in the enhanced image of the first video frame, and the P point is the same as the P point in the enhanced image of the first video frame

S303, determining a corresponding pixel point of the motion point in the target enhanced image aiming at each motion point in the motion point set, and determining the pixel value of the motion point according to the pixel values of a plurality of pixel points in a preset area where the corresponding pixel point is located.

In this embodiment, the corresponding pixel point of the motion point Q in the target enhanced image may refer to the corresponding pixel point of the motion point in the target video frame.

Optionally, the video frame to be processed and the target video frame are adjacent frames, and the coordinates of the corresponding pixel point Q' of the motion point in the target video frame can be determined according to the speed vector of the motion point Q and the coordinates of the motion point in the video frame to be processed.

Wherein the operation speed of the operation point can be obtained by the reference formula (15).

For example, if the coordinate of the motion point Q in the video frame to be processed is (x, y) and the velocity vector is (u _x,v_y), the coordinate of the corresponding pixel point Q' of the motion point Q in the target video frame is (x+u _x×△t,y+v_y ×Δt); i.e. the motion point has coordinates (x+u _x×△t,y+v_y × Δt) at the corresponding pixel point Q' in the target enhanced image.

Wherein Δt is the time interval between the video frame to be processed and the target video frame.

In this embodiment, determining the pixel value of the motion point according to the pixel values of the plurality of pixels in the preset area where the corresponding pixel is located may specifically refer to determining the plurality of pixels in the preset area where the corresponding pixel is located, determining an average value of the pixel values of the plurality of pixels, and taking the average value as the pixel value of the motion point.

The size of the preset area can be adjusted according to the image enhancement effect.

By way of example only, and not by way of limitation,As the pixel value of the motion point Q,For the pixel value of the Q point in the target enhanced image corresponding to the target video frame, thenThe value of (c) may be referred to as (17):

Wherein, (x, y) is the coordinate value of the pixel point Q, and (u _x,v_y) represents the velocity vector of the pixel point Q. (x+u _x×△t,y+v_y × Δt)) is the coordinates of the pixel point corresponding to the pixel point Q in the target video frame I _t, and (x+u _x×△t+i,y+v_y × Δt+j) is the coordinates of a plurality of pixel points in the preset area where the corresponding pixel point is located; 2n is the number of pixel points in the preset area; Δt is the time interval between the video frame to be processed and the target video frame.

For example, taking n as 2, 2n as 4, i.e. determining according to the pixel values of four pixels in the preset region where the corresponding pixel is locatedIs the value of (1):

S304, obtaining an enhanced image of the video frame to be processed according to the pixel values of all the motionless points in the motionless point set and the pixel values of all the motionless points in the motionpoint set.

In the step, according to the pixel values of all the motionless points in the motionless point set and the pixel values of all the motionless points in the motional point set, the pixel values of all the pixel points on the video frame to be processed are adjusted, and the enhanced image of the video frame to be processed is obtained.

For example, for each pixel in a video frame to be processed, if the pixel belongs to a stationary point set, the pixel value of the pixel is adjusted to be the pixel value of the stationary point in the stationary point set, which is the same as the coordinate of the pixel; and if the pixel belongs to the motion point set, adjusting the pixel value of the pixel to be the pixel value of the motion point with the same coordinate as the pixel in the motion point set.

The video frame to be processed includes M pixel points, the coordinate values of the M pixel points are f ₁、、f₂……f_M, the M pixel points are divided into a motionless point set FP and a moving point set DP, the pixel value of each motionless point in the motionless point set FP is predetermined, and the pixel value of each moving point in the moving point set DP is predetermined. Then, for each pixel f _k in the video frame to be processed, where k is greater than or equal to 1 and less than or equal to M, if the pixel belongs to the motionless point set FP, changing the pixel value of the pixel f _k to the pixel value of the motionless point with the coordinate f _k in the motionless point set FP; if the pixel belongs to the motion point set DP, changing the pixel value of the pixel f _k to the pixel value of the motion point with the coordinate f _k in the motion point set DP; and executing the steps on the M pixel points of the video frame to be processed to obtain a changed image, and taking the changed image as an enhanced image of the video frame to be processed. And carrying out the above adjustment on the pixel value of each pixel point in the video frame to be processed to obtain an adjusted video frame to be processed, and taking the adjusted video frame to be processed as an enhanced image of the video frame to be processed.

According to the video enhancement method provided by the embodiment of the application, the pixel points of the video frame to be processed are divided into the motionless point set and the motion point set relative to the target video frame, the pixel values of the motionless points in the motionless point set are the same as the pixel values of the corresponding pixel points in the target enhanced image, and the motion points in the motion point set are updated and obtained by the pixel values of a plurality of pixel points in a preset area where the corresponding pixel points in the target enhanced image are located. The correlation of the pixel value of each pixel point is guaranteed, the space-time consistency of adjacent video frames is further guaranteed, and brightness jitter after the image of the adjacent video frames is enhanced is effectively avoided.

It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.

Based on the video enhancement method provided by the above embodiment, the embodiment of the present invention further provides an apparatus embodiment for implementing the above method embodiment.

Fig. 9 is a schematic structural diagram of a video enhancement device according to an embodiment of the present application. The units included are used to perform the steps in the embodiments corresponding to fig. 1, 3, and 5 to 8, and refer specifically to the relevant descriptions in the embodiments corresponding to fig. 1, 3, and 5 to 8. For convenience of explanation, only the portions related to the present embodiment are shown. Referring to fig. 9, the video enhancement apparatus 60 includes an acquisition module 601, a first enhancement module 602, a first determination module 603, an execution module 604, and a second determination module 605.

The acquiring module 601 is configured to acquire a target video frame in a video to be processed.

The first enhancement module 602 is configured to perform image enhancement processing on the target video frame, so as to obtain a target enhanced image corresponding to the target video frame.

A first determining module 603, configured to determine an enhanced image corresponding to the video frame to be processed according to the target enhanced image; the video frame to be processed is a video frame to be subjected to image enhancement processing, which is determined according to the target video frame.

The executing module 604 is configured to take the video frame to be processed as a target video frame, and take an enhanced image corresponding to the video frame to be processed as a target enhanced image, and return to execute the step of determining the enhanced image corresponding to the video frame to be processed according to the target enhanced image until the enhanced images respectively corresponding to the preset number of video frames in the video to be processed are obtained.

The second determining module 605 is configured to determine an enhanced video corresponding to the video to be processed according to the enhanced image of the video frame in the video to be processed.

Optionally, the video frame to be processed is a video frame adjacent to the target video frame.

Optionally, the image enhancement processing is performed on the target video frame by the first enhancement module 602 to obtain a target enhanced image corresponding to the target video frame, including:

acquiring a brightness channel of a target video frame;

determining a plurality of illumination components of the target video frame according to the brightness channel of the target video frame;

Determining a fusion illumination component corresponding to the target video frame according to the illumination components of the target video frame;

And processing the target video frame according to the fusion illumination component to generate a target enhanced image corresponding to the target video frame.

Optionally, the first enhancement module 602 determines a plurality of illumination components of the target video frame according to the luminance channel of the target video frame, including:

performing low-pass filtering treatment on the brightness channel to obtain a first illumination component;

and performing a variation operation on the brightness channel to obtain a second illumination component.

Optionally, the first enhancement module 602 determines, according to the multiple illumination components of the target video frame, a fused illumination component corresponding to the target video frame, including:

Determining a first gradient of the first illumination component and a second gradient of the second illumination component;

and based on the first gradient and the second gradient, carrying out fusion processing on the first illumination component and the second illumination component to obtain a fusion illumination component.

Optionally, the first enhancement module 602 processes the target video frame according to the fused illumination component, and generates a target enhanced image corresponding to the target video frame, including:

Determining a reflection component of the luminance channel according to the fused illumination component and the luminance channel;

Correcting the reflection component to generate a corrected reflection component;

Nonlinear stretching is carried out on the fusion illumination component, and a stretched fusion illumination component is generated;

obtaining an enhanced brightness channel according to the stretched fusion illumination component and the corrected reflection component;

And processing the target video frame according to the enhanced brightness channel to generate a target enhanced image corresponding to the target video frame.

Optionally, the first enhancement module 602 processes the target video frame according to the enhanced brightness channel, and generates a target enhanced image corresponding to the target video frame, including:

Acquiring a saturation channel and a chromaticity channel of a target video frame;

Generating an RGB color image according to the enhanced brightness channel, the saturation channel and the chromaticity channel;

and taking the RGB color image as a target enhancement image corresponding to the target video frame.

Optionally, the first enhancement module 602 generates an RGB color image according to the enhanced luminance channel, the saturation channel, and the chrominance channel, including:

Dividing the enhanced brightness channel into a plurality of image blocks;

For each image block, carrying out contrast enhancement processing on the brightness of the image block to obtain an enhanced image block corresponding to the image block;

splicing adjacent enhanced image blocks to obtain an updated enhanced brightness channel;

And generating an RGB color image according to the updated enhanced brightness channel, the saturation channel and the chromaticity channel.

Optionally, the first determining module 603 determines, according to the target enhanced image, an enhanced image corresponding to the video frame to be processed, including:

determining a fixed point set and a moving point set of a pixel point of a video frame to be processed relative to a target video frame;

For each stationary point in the stationary point set, taking the pixel value of the pixel point corresponding to the stationary point in the target enhanced image as the pixel value of the stationary point;

determining corresponding pixel points of the motion points in the target enhanced image aiming at each motion point in the motion point set, and determining pixel values of the motion points according to pixel values of a plurality of pixel points in a preset area where the corresponding pixel points are located;

And obtaining an enhanced image of the video frame to be processed according to the pixel values of all the motionless points in the motionless point set and the pixel values of all the motionless points in the motionpoint set.

Optionally, the first determining module 603 determines a set of motionless points and a set of motioning points of pixels of the video frame to be processed with respect to the target video frame, including:

acquiring an optical flow vector of a pixel point aiming at each pixel point in a video frame to be processed, and determining the motion speed of the pixel point according to the optical flow vector of the pixel point;

Determining pixel points with the motion speed greater than a preset value as motion points, and generating a motion point set according to all the motion points;

And determining the pixel points with the motion speed smaller than or equal to a preset value as the motionless points, and generating a motionless point set according to all motionless points.

Optionally, the first determining module 603 determines the pixel value of the motion point according to the pixel values of the plurality of pixel points in the preset area where the corresponding pixel point is located, including:

Determining a plurality of pixel points in a preset area where the corresponding pixel points are located;

Determining an average value of pixel values of a plurality of pixel points;

The average value is taken as the pixel value of the motion point.

Fig. 10 is a schematic diagram of a video processing apparatus according to an embodiment of the present application. As shown in fig. 10, the video processing apparatus 70 of this embodiment includes: at least one processor 701, a memory 702 and a computer program stored in the memory 702 and executable on the processor 701. The video processing device further comprises communication means 703, wherein the processor 701, the memory 702 and the communication means 703 are connected by a bus 704.

The steps in the various video enhancement method embodiments described above, such as steps S10 through S50 in the embodiment shown in fig. 1, are implemented by the processor 701 when executing the computer program. Or the processor 701, when executing a computer program, performs the functions of the modules/units in the above-described apparatus embodiments, for example, the functions of the modules 601 to 605 shown in fig. 9.

It will be appreciated by those skilled in the art that fig. 10 is merely an example of a video processing device and is not intended to limit the video processing device, and may include more or fewer components than shown, or may combine certain components, or different components, such as input-output devices, network access devices, buses, etc.

The Processor 701 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 702 may be an internal storage unit of the video processing device or an external storage device of the video processing device, and the memory 702 is used for storing the computer program and other programs and data required by the video processing device. The memory 702 may also be used to temporarily store data that has been output or is to be output.

Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps for implementing the various method embodiments described above.

Embodiments of the present application provide a computer program product which, when run on a mobile terminal, causes the mobile terminal to perform steps that enable the implementation of the method embodiments described above.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above-described embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of the method embodiments described above. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, executable files or in some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/terminal apparatus, recording medium, computer Memory, read-Only Memory (ROM), random access Memory (RAM, random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other manners. The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims (12)

1. A method of video enhancement, comprising:

acquiring a target video frame in a video to be processed, and performing image enhancement processing on the target video frame to obtain a target enhanced image corresponding to the target video frame;

Determining an enhanced image corresponding to the video frame to be processed according to the target enhanced image, wherein the method comprises the following steps: determining a fixed point set and a moving point set of pixel points of the video frame to be processed relative to the target video frame; for each stationary point in the stationary point set, taking a pixel value of a pixel point corresponding to the stationary point in the target enhanced image as the pixel value of the stationary point; determining a corresponding pixel point of the motion point in the target enhanced image aiming at each motion point in the motion point set, and determining the pixel value of the motion point according to the pixel values of a plurality of pixel points in a preset area where the corresponding pixel point is located; obtaining an enhanced image of the video frame to be processed according to the pixel values of all the motionless points in the motionless point set and the pixel values of all the motionless points in the motionpoint set; the video frame to be processed is a video frame to be subjected to image enhancement processing, which is determined according to the target video frame;

Taking the video frame to be processed as a target video frame, taking an enhanced image corresponding to the video frame to be processed as a target enhanced image, and returning to execute the step of determining the enhanced image corresponding to the video frame to be processed according to the target enhanced image until the enhanced images respectively corresponding to the preset number of video frames in the video to be processed are obtained;

and determining the enhanced video corresponding to the video to be processed according to the enhanced image corresponding to the video frame in the video to be processed.

2. The video enhancement method of claim 1, wherein the video frame to be processed is a video frame adjacent to the target video frame.

3. The video enhancement method according to claim 1, wherein the performing image enhancement processing on the target video frame to obtain a target enhanced image corresponding to the target video frame includes:

Acquiring a brightness channel of the target video frame;

Determining a plurality of illumination components of the target video frame according to the brightness channel of the target video frame;

Determining a fusion illumination component corresponding to the target video frame according to the illumination components of the target video frame;

and processing the target video frame according to the fusion illumination component to generate a target enhanced image corresponding to the target video frame.

4. The video enhancement method of claim 3, wherein said determining a plurality of illumination components of said target video frame from a luminance channel of said target video frame comprises:

performing low-pass filtering processing on the brightness channel to obtain a first illumination component;

And performing variation operation on the brightness channel to obtain a second illumination component.

5. The video enhancement method of claim 4, wherein said determining a corresponding blended illumination component for said target video frame from a plurality of illumination components for said target video frame comprises:

determining a first gradient of the first illumination component and a second gradient of the second illumination component;

and based on the first gradient and the second gradient, carrying out fusion processing on the first illumination component and the second illumination component to obtain the fusion illumination component.

6. A video enhancement method according to claim 3, wherein said processing said target video frame according to said fused illumination component to generate a target enhanced image corresponding to said target video frame comprises:

Determining a reflection component of the luminance channel according to the fused illumination component and the luminance channel;

correcting the reflection component to generate a corrected reflection component;

nonlinear stretching is carried out on the fusion illumination component, and a stretched fusion illumination component is generated;

obtaining an enhanced brightness channel according to the stretched fusion illumination component and the corrected reflection component;

and processing the target video frame according to the enhanced brightness channel to generate a target enhanced image corresponding to the target video frame.

7. The video enhancement method according to claim 6, wherein said processing the target video frame according to the enhanced luminance channel to generate a target enhanced image corresponding to the target video frame comprises:

acquiring a saturation channel and a chromaticity channel of the target video frame;

Generating an RGB color image according to the enhanced brightness channel, the saturation channel and the chromaticity channel;

and taking the RGB color image as a target enhancement image corresponding to the target video frame.

8. The video enhancement method of claim 7, wherein said generating an RGB color image from said enhanced luminance channel, said saturation channel, and said chrominance channel comprises:

dividing the enhanced brightness channel into a plurality of image blocks;

For each image block, carrying out contrast enhancement processing on the brightness of the image block to obtain an enhanced image block corresponding to the image block;

splicing adjacent enhanced image blocks to obtain an updated enhanced brightness channel;

And generating the RGB color image according to the updated enhanced brightness channel, the saturation channel and the chromaticity channel.

9. The video enhancement method of claim 1, wherein the determining the set of stationary points and the set of moving points in the pixels of the video frame to be processed relative to the target video frame comprises:

for each pixel point in the video frame to be processed, acquiring an optical flow vector of the pixel point, and determining the motion speed of the pixel point according to the optical flow vector of the pixel point;

Determining pixel points with the motion speed greater than a preset value as motion points, and determining the motion point set according to all the motion points;

And determining the pixel points with the motion speed smaller than or equal to the preset value as the motionless points, and determining the motionless point set according to all motionless points.

10. The video enhancement method according to claim 1, wherein the determining the pixel value of the motion point according to the pixel values of a plurality of pixel points in a preset area where the corresponding pixel point is located includes:

Determining a plurality of pixel points in a preset area where the corresponding pixel points are located;

determining an average value of pixel values of the plurality of pixel points;

And taking the average value as the pixel value of the motion point.

11. A video processing device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 10 when the computer program is executed.

12. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 10.