CN116957953A

CN116957953A - Brightness enhancement method and training method for enhancement parameter perception model

Info

Publication number: CN116957953A
Application number: CN202310421858.4A
Authority: CN
Inventors: 彭程威; 李峰; 左小祥
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-04-18
Filing date: 2023-04-18
Publication date: 2023-10-27

Abstract

The embodiment of the application discloses a brightness enhancement method and a training method for enhancing a parameter perception model, and belongs to the technical field of computers. The embodiment of the application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving, image recognition and the like. The method comprises the following steps: acquiring a brightness image of a first video frame in a video; extracting brightness features of a brightness image based on an enhancement parameter perception model, mapping the brightness features into brightness enhancement parameters in a brightness enhancement function, wherein the enhancement parameter perception model is used for determining brightness enhancement parameters of the input image in the brightness enhancement function based on brightness values of a plurality of pixel points in the input image, and the brightness enhancement function is used for determining brightness values of the pixel points after the pixel points are enhanced according to the brightness enhancement parameters; and based on the brightness enhancement parameters, enhancing the brightness values of the pixel points in the brightness image through a brightness enhancement function to obtain a second video frame. The scheme can improve the brightness enhancement effect of the video frame.

Description

Brightness enhancement method and training method for enhancement parameter perception model

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a brightness enhancement method and a training method for enhancing a parameter perception model.

Background

With the development of computer technology, online video chat and online video conference gradually become common communication modes in real life. In the process that the user participates in video chat, the brightness of a video picture corresponding to the user is low due to poor light of the environment where the user is located, and the quality of online video is affected. Therefore, how to enhance the brightness of video frames is a technical problem to be solved.

In the related art, a unified brightness enhancement algorithm is generally adopted to enhance the brightness of video frames in video. In addition, in the process of enhancing the brightness of the video frames, the brightness enhancement force of the video frames with different brightness is the same.

Therefore, the method can lead to brighter video frames in the video after being enhanced, and reduces the display effect of the video.

Disclosure of Invention

The embodiment of the application provides a brightness enhancement method and a training method for enhancing a parameter perception model, which can improve the brightness enhancement effect of a video frame. The technical scheme is as follows:

in one aspect, there is provided a brightness enhancement method, the method comprising:

acquiring a brightness image of a first video frame in a video, wherein the brightness image is an image of a brightness channel of the first video frame in a YUV space;

Extracting brightness characteristics of the brightness image based on an enhancement parameter perception model, mapping the brightness characteristics into brightness enhancement parameters in a brightness enhancement function, wherein the enhancement parameter perception model is used for determining the brightness enhancement parameters of the input image in the brightness enhancement function based on brightness values of a plurality of pixel points in the input image, and the brightness enhancement function is used for determining the brightness values of the pixel points after the pixel points are enhanced according to the brightness enhancement parameters;

and based on the brightness enhancement parameters, enhancing the brightness values of the pixel points in the brightness image through the brightness enhancement function to obtain a second video frame, wherein the brightness of the second video frame is larger than that of the first video frame.

In one aspect, a training method for enhancing a parameter perception model is provided, the method comprising:

acquiring a brightness image of a sample image and a label image of the sample image, wherein the label image is an image obtained by enhancing the brightness of the sample image;

extracting brightness characteristics of the brightness image based on an enhancement parameter perception model, mapping the brightness characteristics into brightness enhancement parameters in a brightness enhancement function, wherein the enhancement parameter perception model is used for determining the brightness enhancement parameters of the input image in the brightness enhancement function based on brightness values of a plurality of pixel points in the input image, and the brightness enhancement parameter function is used for determining the brightness values of the pixel points after the pixel points are enhanced according to the brightness enhancement parameters;

Based on the brightness enhancement parameters, enhancing brightness values of pixel points in the brightness image through the brightness enhancement function to obtain a sample image with enhanced brightness;

determining a training loss of the enhanced parameter perception model based on the brightness enhanced sample image and the label image;

based on the training loss, training the enhanced parameter perception model.

In another aspect, there is provided a brightness enhancement device, the device comprising:

the acquisition module is used for acquiring a brightness image of a first video frame in a video, wherein the brightness image is an image of a brightness channel of the first video frame in a YUV space;

the brightness enhancement module is used for extracting brightness features of the brightness image based on an enhancement parameter perception model, mapping the brightness features into brightness enhancement parameters in a brightness enhancement function, wherein the enhancement parameter perception model is used for determining brightness enhancement parameters of the input image in the brightness enhancement function based on brightness values of a plurality of pixel points in the input image, and the brightness enhancement function is used for determining brightness values of the pixel points after the pixel points are enhanced according to the brightness enhancement parameters;

and the brightness enhancement module is used for enhancing the brightness value of the pixel point in the brightness image through the brightness enhancement function based on the brightness enhancement parameter to obtain a second video frame, and the brightness of the second video frame is larger than that of the first video frame.

In some embodiments, the brightness enhancement module is configured to exponentially moving average brightness enhancement parameters of a brightness image of the first video frame based on a smooth brightness enhancement parameter of a third video frame, where the third video frame is a previous video frame of the first video frame, and the smooth brightness enhancement parameter of the first video frame in the video is the brightness enhancement parameter of the first video frame; and based on the smooth brightness enhancement parameters, enhancing the brightness values of pixel points in the brightness image of the first video frame through the brightness enhancement function to obtain the second video frame.

In some embodiments, the brightness enhancement module is configured to perform exponential moving average on brightness enhancement parameters of brightness images of the first video frame based on brightness enhancement parameters of N fourth video frames, where N is a positive integer, and a frame number of the fourth video frame is smaller than a frame number of the first video frame; and based on the smooth brightness enhancement parameters, enhancing the brightness values of pixel points in the brightness image of the first video frame through the brightness enhancement function to obtain the second video frame.

In some embodiments, the brightness enhancement module comprises:

a determining unit, configured to determine, for any pixel point in the luminance image, a luminance threshold value corresponding to a current luminance value of the pixel point in a luminance lookup table, where the luminance lookup table is used to indicate a correspondence between the luminance value of the pixel point and the luminance threshold value, and the luminance threshold value is used to indicate an upper limit value of the luminance value of the pixel point;

the brightness enhancement unit is used for processing the brightness threshold value of the pixel point and the current brightness value of the pixel point through the brightness enhancement function based on the smooth brightness enhancement parameter to obtain the brightness value of the pixel point after enhancement;

and the generating unit is used for generating the second video frame based on the brightness values after the pixel points are enhanced.

In some embodiments, the brightness enhancement unit is configured to determine, based on the smooth brightness enhancement parameter, a first coefficient and a second coefficient in the brightness enhancement function, where the first coefficient is a coefficient of a brightness threshold of the pixel, the first coefficient is positively related to the smooth brightness enhancement parameter, the second coefficient is a coefficient of a current brightness value of the pixel, and the second coefficient is negatively related to the smooth brightness enhancement parameter; based on a first coefficient and a second coefficient in the brightness enhancement function, respectively processing a brightness threshold value of the pixel point and a current brightness value of the pixel point to obtain a first brightness value and a second brightness value, wherein the first brightness value is the product of the first coefficient and the brightness threshold value of the pixel point, and the second brightness value is the product of the second coefficient and the current brightness value of the pixel point; and taking the sum of the first brightness value and the second brightness value as the brightness value after the pixel point is enhanced.

In some embodiments, the enhanced parameter perception model comprises a depth separable convolutional layer comprising a depth convolutional layer and a point-by-point convolutional layer;

the determining module is configured to convolve luminance values of a plurality of pixel points in the luminance image based on a depth convolution layer in the depth separable convolution layers to obtain an intermediate luminance feature of the luminance image, where the depth of the intermediate luminance feature is 1; and convolving the intermediate brightness characteristic based on a point-by-point convolution layer in the depth separable convolution layer to obtain the brightness characteristic of the brightness image, wherein the depth of the brightness characteristic is the same as the number of output channels of a convolution kernel in the point-by-point convolution layer.

In some embodiments, the enhanced parameter perception model further comprises a fully connected layer and an activation function layer;

the determining module is used for processing the brightness characteristic based on the full connection layer to obtain a one-dimensional characteristic vector; the one-dimensional feature vector is mapped to the luminance enhancement parameter based on the activation function layer.

In some embodiments, the brightness enhancement module is configured to determine, based on the brightness enhancement parameter, a brightness value of the brightness image after the pixel point is enhanced by the brightness enhancement function; generating an intermediate brightness image based on the brightness value of the enhanced pixel point in the brightness image; and fusing the intermediate brightness image, the image of the U channel of the first video frame in the YUV space and the image of the V channel of the first video frame in the YUV space to obtain the second video frame, wherein the second video frame is an RGB image.

In some embodiments, the apparatus further comprises:

the normalization module is used for normalizing the brightness values of a plurality of pixel points in the brightness image;

and the inverse normalization module is used for carrying out inverse normalization on the brightness values of a plurality of pixel points in the intermediate brightness image to obtain integer brightness values of the plurality of pixel points.

In some embodiments, the apparatus further comprises:

and the downsampling module is used for downsampling the brightness image.

In another aspect, a training apparatus for enhancing a parametric perceptual model is provided, the apparatus comprising:

the acquisition module is used for acquiring a brightness image of a sample image and a label image of the sample image, wherein the label image is an image obtained by enhancing the brightness of the sample image;

The brightness enhancement module is used for enhancing the brightness value of the pixel point in the brightness image through the brightness enhancement function based on the brightness enhancement parameter to obtain a sample image with enhanced brightness;

the training loss determination module is used for determining the training loss of the enhanced parameter perception model based on the sample image with enhanced brightness and the label image;

and the training module is used for training the enhanced parameter perception model based on the training loss.

In another aspect, a computer device is provided, the computer device including a processor and a memory for storing at least one segment of a computer program loaded and executed by the processor to implement a brightness enhancement method in an embodiment of the application.

In another aspect, a computer device is provided, the computer device including a processor and a memory for storing at least one segment of a computer program loaded and executed by the processor to implement a training method for an enhanced parameter awareness model in an embodiment of the application.

In another aspect, a computer readable storage medium having stored therein at least one segment of a computer program loaded and executed by a processor to implement a brightness enhancement method as in an embodiment of the present application is provided.

In another aspect, a computer readable storage medium is provided, in which at least one segment of a computer program is stored, the at least one segment of the computer program being loaded and executed by a processor to implement a training method for enhancing a parametric perceptual model in an embodiment of the present application.

In another aspect, a computer program product is provided, comprising a computer program that is executed by a processor to implement the brightness enhancement method provided in an embodiment of the application.

In another aspect, a computer program product is provided, comprising a computer program that is executed by a processor to implement a training method for enhancing a parametric perceptual model in an embodiment of the present application.

The embodiment of the application provides a brightness enhancement method. In the process of enhancing the brightness of a video frame in a video, a brightness image of the video frame in a brightness channel is obtained. And extracting the brightness characteristics of the brightness image according to the current brightness value of each pixel point in the brightness image by the enhanced parameter perception model. The enhancement parameter perception model adaptively determines a luminance enhancement parameter of a luminance image in a luminance enhancement function according to a luminance characteristic of the luminance image. The brightness enhancement function can determine the brightness value of the pixel after the pixel is enhanced according to the current brightness value of the pixel, and the brightness enhancement parameter is used for determining the enhancement strength of the brightness value of the pixel. Therefore, the enhancement parameter perception model can adaptively perceive the enhancement parameters of different video frames, namely the brightness enhancement strength of different video frames according to the brightness characteristics of different video frames. Compared with the mode of reinforcing all video frames by adopting fixed reinforcing force, the processing mode avoids the problem that part of video frames are too dark or too bright, and improves the brightness reinforcing effect of the video frames.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic illustration of an implementation environment provided by an embodiment of the present application;

FIG. 2 is a flowchart of a brightness enhancement method according to an embodiment of the present application;

FIG. 3 is a flowchart of another brightness enhancement method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an enhanced parameter perception model provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a brightness enhancement curve according to an embodiment of the present application;

FIG. 6 is a flow chart of a brightness enhancement provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a video frame provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of another video frame provided by an embodiment of the present application;

FIG. 9 is a flowchart of a training method for enhancing a parameter perception model according to an embodiment of the present application;

Fig. 10 is a schematic structural diagram of a brightness enhancement device according to an embodiment of the present application;

fig. 11 is a schematic structural view of another brightness enhancement device according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a training device for enhancing a parameter perception model according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of a terminal according to an embodiment of the present application;

fig. 14 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the following detailed description of the embodiments of the present application will be given with reference to the accompanying drawings.

The terms "first," "second," and the like in this disclosure are used for distinguishing between similar elements or items having substantially the same function and function, and it should be understood that there is no logical or chronological dependency between the terms "first," "second," and "n," and that there is no limitation on the amount and order of execution.

The term "at least one" in the present application means one or more, and the meaning of "a plurality of" means two or more.

It should be noted that, the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, presented data, etc.), and signals related to the present application are all authorized by the user or are fully authorized by the parties, and the collection, use, and processing of the related data is required to comply with the relevant laws and regulations and standards of the relevant countries and regions. For example, the video and the first video frame in the video referred to in the present application are acquired with sufficient authorization.

The following terms are used in connection with the present application:

cloud computing (clouding) is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to acquire computing power, storage space, and information services as needed. The network that provides the resources is referred to as the "cloud". Resources in the cloud are infinitely expandable in the sense of users, and can be acquired at any time, used as needed, expanded at any time and paid for use as needed. As a basic capability provider of cloud computing, a cloud computing resource pool (abbreviated as a cloud platform, generally called IaaS (Infrastructure as a Service, infrastructure as a service) platform) is established, and multiple types of virtual resources are deployed in the resource pool for external clients to select for use. The cloud computing resource pool mainly comprises: computing devices (which are virtualized machines, including operating systems), storage devices, network devices. According to the logic function division, a PaaS (Platform as a Service ) layer can be deployed on the IaS layer, a SaaS (Software as a Service ) layer can be deployed on the PaaS layer, and the SaaS can also be directly deployed on the IaS. PaaS is a platform for software running, such as a database, a web (world wide web) container, etc. SaaS is a wide variety of business software such as web portals, sms mass senders, etc. Generally, saaS and PaaS are upper layers relative to IaaS.

Cloud conferencing is an efficient, convenient, low-cost form of conferencing based on cloud computing technology. The user can rapidly and efficiently share voice, data files and videos with all groups and clients in the world synchronously by simply and easily operating through an internet interface, and the user is helped by a cloud conference service provider to operate through complex technologies such as data transmission, processing and the like in the conference. At present, domestic cloud conference mainly focuses on service contents mainly in a SaaS (Software as a Service ) mode, including service forms of telephone, network, video and the like, and video conference based on cloud computing is called as a cloud conference. In the cloud conference era, the transmission, processing and storage of data are all processed by the computer resources of video conference factories, and users can carry out efficient remote conferences without purchasing expensive hardware and installing complicated software. The cloud conference system supports the dynamic cluster deployment of multiple servers, provides multiple high-performance servers, and greatly improves conference stability, safety and usability. In recent years, video conferences are popular for a plurality of users because of greatly improving communication efficiency, continuously reducing communication cost and bringing about upgrade of internal management level, and have been widely used in various fields of transportation, finance, operators, education, enterprises and the like. Undoubtedly, the video conference has stronger attraction in convenience, rapidness and usability after the cloud computing is applied, and the video conference application is required to be stimulated.

The RGB color space is a way to describe colors, and includes three color channels of R (Red), G (Green), and B (Blue). In the RGB color space, each color may be superimposed by three basic colors of different degrees. The present application is simply referred to as RGB space.

The YUV color space is a way to describe colors, in which each pixel consists of three components, luminance Y, chrominance U, and chrominance V, respectively. Wherein the luminance component is used to represent luminance information of the image, and the chrominance U and the chrominance V are used to represent color information of the image. The application is simply referred to as YUV space.

FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application, referring to FIG. 1, the implementation environment includes: a terminal 101 and a server 102. The terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.

In some embodiments, terminal 101 is, but is not limited to, a smart phone, tablet, notebook, desktop, smart voice interaction device, smart home appliance, in-vehicle terminal, aircraft, etc. The terminal 101 installs and runs various applications supporting any one of functions of video recording, video distribution, and video processing, such as a video conference type application, a video live broadcast type application, and a video sharing type application. Applications are used to process video, such as enhancing the brightness of video frames in video. After the terminal 101 enhances the brightness of the video frame in the video through the application program, the terminal 101 may locally display the video frame after the brightness enhancement. The terminal 101 may also transmit the video frame with enhanced brightness to the server 102, and the server 102 may transmit the video frame with enhanced brightness to other terminals. In some embodiments, the terminal 101 may send the brightness enhanced video stream to the server 102. The video stream includes a plurality of luminance enhanced video frames. By the method, the brightness of the video picture corresponding to the user can be enhanced in the process of carrying out the multi-person video conference through the video conference application program.

In some embodiments, the server 102 is a stand-alone physical server, can be a server cluster or a distributed system formed by a plurality of physical servers, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), and basic cloud computing services such as big data and artificial intelligence platforms. In some embodiments, server 102 receives the brightness enhanced video frames uploaded by terminal 101 through the application. The server 102 transmits the video frame with enhanced brightness to other terminals. And displaying the video frames with enhanced brightness by other terminals through application programs. In some embodiments, the server 102 takes on primary computing work and the terminal 101 takes on secondary computing work; alternatively, the server 102 takes on secondary computing work and the terminal 101 takes on primary computing work; alternatively, a distributed computing architecture is used for collaborative computing between the server 102 and the terminal 101.

Those skilled in the art will recognize that the number of terminals may be greater or lesser. Such as the above-mentioned terminals may be only one, or the above-mentioned terminals may be several tens or hundreds, or more. The embodiment of the application does not limit the number of terminals and the equipment type.

Fig. 2 is a flowchart of a brightness enhancement method according to an embodiment of the present application, and as shown in fig. 2, the brightness enhancement method is described by way of example in the embodiment of the present application. The brightness enhancement method comprises the following steps:

201. the terminal acquires a brightness image of a first video frame in the video, wherein the brightness image is an image of a brightness channel of the first video frame in a YUV space.

In the embodiment of the application, the brightness of the video frames in the video is lower due to poor light of the recording environment in the video recording process, so that the video content is difficult to see. It is therefore necessary to appropriately enhance the brightness of video to improve the quality of video. The video is a video with brightness to be enhanced, and the video comprises a plurality of video frames with brightness to be enhanced. The terminal determines a first video frame from a plurality of video frames in the video to be enhanced in brightness. For example, the terminal may take any one of the video frames as the first video frame. The brightness of the first video frame is low. The image format of the first video frame may be RGB (Red, green, blue, red, green, blue) format or YUV format, which is not limited in the embodiment of the present application. Taking the image format of the first video frame as an RGB format as an example, the terminal converts the first video frame from an RGB space to a YUV space (YUV color space) to obtain the first video frame in YUV format. The terminal separates Y channels, namely brightness channels, in the first video frame in YUV format to obtain a brightness image of the first video frame. The luminance image is an image having a luminance value of a pixel point as a pixel value.

202. The terminal extracts brightness characteristics of a brightness image based on an enhancement parameter perception model, maps the brightness characteristics to brightness enhancement parameters in a brightness enhancement function, wherein the enhancement parameter perception model is used for determining brightness enhancement parameters of the input image in the brightness enhancement function based on brightness values of a plurality of pixel points in the input image, and the brightness enhancement function is used for determining brightness values of the pixel points after the pixel points are enhanced according to the brightness enhancement parameters.

In the embodiment of the application, the terminal processes the brightness values of a plurality of pixel points in the brightness image of the first video frame through the enhanced parameter perception model, so that the brightness characteristics of the brightness image can be obtained. After the brightness characteristics of the brightness image are determined by the enhancement parameter perception model, the brightness characteristics are mapped into brightness enhancement parameters in a brightness enhancement function. The brightness enhancement function is used for determining a brightness value of the pixel point in the brightness image after the pixel point in the brightness image is enhanced according to the brightness enhancement parameter, and the brightness enhancement parameter can indicate the brightness enhancement strength of the pixel point in the brightness image. For example, the larger the brightness enhancement parameter is, the larger the brightness enhancement strength of the pixel point is, and the larger the brightness value of the pixel point after enhancement is determined by the brightness enhancement function is; the smaller the brightness enhancement parameter is, the smaller the brightness enhancement strength of the pixel point is, and the smaller the brightness value of the pixel point after being enhanced is determined by the brightness enhancement function. Therefore, through the enhancement parameter perception model, different enhancement parameters can be determined according to the brightness characteristics of the brightness images of different video frames, namely different brightness enhancement forces are determined for the brightness images of different video frames.

203. And the terminal enhances the brightness value of the pixel point in the brightness image through a brightness enhancement function based on the brightness enhancement parameter to obtain a second video frame, wherein the brightness of the second video frame is larger than that of the first video frame.

In the embodiment of the application, after the terminal determines the brightness enhancement parameters in the brightness enhancement function, the terminal processes the current brightness value of the pixel point in the brightness image through the brightness enhancement function and the brightness enhancement parameters in the brightness enhancement function to obtain the brightness value after the pixel point is enhanced. And the terminal adjusts the brightness value of the pixel point in the brightness image to be the brightness value after the pixel point is enhanced, so as to obtain the enhanced brightness image. And the terminal converts the enhanced brightness image into an RGB space to obtain a second video frame in an RGB format. The second video frame is a video frame after the terminal carries out brightness enhancement on the first video frame, and the brightness of the second video frame is larger than that of the first video frame.

The embodiment of the application provides a brightness enhancement method. In the process of enhancing the brightness of a video frame in a video, a brightness image of the video frame in a brightness channel is obtained. And extracting the brightness characteristics of the brightness image according to the current brightness value of each pixel point in the brightness image by the enhanced parameter perception model. The enhancement parameter perception model adaptively determines a luminance enhancement parameter of a luminance image in a luminance enhancement function according to a luminance characteristic of the luminance image. The brightness enhancement function can determine the brightness value of the pixel after the pixel is enhanced according to the current brightness value of the pixel, and the brightness enhancement parameter is used for determining the brightness enhancement strength of the brightness value of the pixel. Therefore, the enhancement parameter perception model can adaptively perceive the enhancement parameters of different video frames, namely the brightness enhancement strength of different video frames according to the brightness characteristics of different video frames. Compared with the mode of reinforcing all video frames by adopting fixed reinforcing force, the processing mode avoids the problem that part of video frames are too dark or too bright, and improves the brightness reinforcing effect of the video frames.

Fig. 3 is a flowchart of another brightness enhancement method according to an embodiment of the present application, as shown in fig. 3, and in the embodiment of the present application, an example of the brightness enhancement method is described by a terminal. The brightness enhancement method comprises the following steps:

301. the terminal acquires a brightness image of a first video frame in the video, wherein the brightness image is an image of a brightness channel of the first video frame in a YUV space.

In the embodiment of the application, various application programs such as a video conference application program, a video live broadcast application program, a video sharing application program and the like have the functions of video recording or video publishing. In the video recording process, due to poor light of the recording environment, the brightness of video frames in the recorded video is too low, the video content is difficult to see clearly, and the quality of the video is low. Therefore, it is necessary to enhance the brightness of the recorded video, and properly enhance the brightness of a plurality of video frames in the video, so as to enhance the quality of the video. The terminal determines a first video frame from a plurality of video frames of the video to be processed. For example, the terminal may take any one of the video frames as the first video frame. The terminal can acquire the video to be processed through various application programs such as a video conference application program, a video live broadcast application program or a video sharing application program.

The image format of the first video frame may be an RGB format or a YUV format, which is not limited in the embodiment of the present application. Taking the image format of the first video frame as an example of an RGB format, the terminal converts the first video frame from the RGB space to the YUV space to obtain the first video frame in the YUV format. The terminal separates Y channels, namely brightness channels, in the first video frame in YUV format to obtain a brightness image of the first video frame. The luminance image is an image having a luminance value of a pixel point as a pixel value.

In some embodiments, the terminal can pre-process the luminance image after it acquires the luminance image. For example, the terminal downsamples the luminance image. Wherein the terminal may downsample the luminance image by a downsampling function, such as a bilinear interpolation function. The terminal may also downsample the luminance image through a max-pooling operation or an average pooling operation, which is not limited by the embodiment of the present application. By downsampling the luminance image, the number of pixels in the luminance image can be reduced, reducing the resolution of the luminance image. Processing the luminance image with fewer pixel points and lower resolution after downsampling can improve the processing efficiency.

302. The terminal extracts the brightness characteristics of the brightness image based on an enhancement parameter perception model, wherein the enhancement parameter perception model is used for determining brightness enhancement parameters of the input image in a brightness enhancement function based on brightness values of a plurality of pixel points in the input image, and the brightness enhancement function is used for determining brightness values of the pixel points after the pixel points are enhanced according to the brightness enhancement parameters.

In the embodiment of the application, the enhanced parameter perception model can extract the brightness characteristics of the brightness image. And the terminal processes the brightness values of a plurality of pixel points in the brightness image of the first video frame through the enhanced parameter perception model to obtain the brightness characteristics of the brightness image. The brightness characteristic can indicate the degree of brightness of the brightness image. The enhancement parameter perception model is also used for determining the brightness enhancement strength of the brightness image according to the brightness characteristics, namely determining the brightness enhancement parameters of the brightness image in the brightness enhancement function.

In some embodiments, the terminal convolves the luminance image with the enhanced parameter perception model to obtain luminance characteristics of the luminance image. The enhanced parametric perceptual model comprises a depth separable convolutional layer comprising a concatenated depth convolutional layer and a point-by-point convolutional layer. The depth convolution layer is used for convolving the input image without changing the depth of the input image to obtain the characteristics of the input image. The point-by-point convolution layer convolves the image features output by the depth convolution layer by taking each pixel point as a minimum unit, and the point-by-point convolution layer can increase or decrease the depth of the image features. The terminal convolves the brightness values of a plurality of pixel points in the brightness image based on the depth convolution layer in the depth separable convolution layer to obtain the middle brightness characteristic of the brightness image. Since the input of the depth convolution layer is a luminance image of a single channel and the depth is not changed after the convolution of the depth convolution layer, the depth of the intermediate luminance feature output by the depth convolution layer is 1, that is, the number of channels output is 1. The terminal convolves the intermediate brightness characteristic based on a point-by-point convolution layer in the depth separable convolution layers to obtain the brightness characteristic of the brightness image. The depth of the luminance feature is the same as the number of output channels of the convolution kernel in the point-wise convolution layer. For example, in the case where the number of output channels of the convolution kernel in the point-by-point convolution layer is 16, the depth of the luminance feature output by the point-by-point convolution layer is 16. The depth separable convolution layer in the enhanced parameter perception model is used for convoluting the brightness image of the video frame in the brightness channel to obtain the brightness characteristic of the brightness image, so that the calculated amount can be remarkably reduced, and the processing efficiency of the video frame is improved.

For example, the terminal convolves a 12×12×1 luminance image in a depth convolution layer with a convolution kernel of size 3*3 and step size 2 to obtain a 4×4×1 intermediate luminance feature. Then the terminal convolves the intermediate luminance characteristics of 4 x 1 in the point-by-point convolution layer by 16 convolution kernels with a size of 1*1 and a step of 1, so as to obtain the luminance characteristics of 4 x 16. Therefore, by processing the intermediate luminance feature with the depth of 1 by 16 convolution kernels (the number of output channels of the convolution kernels is 16) of the point-by-point convolution layer, the luminance feature with the depth of 16 is obtained, and the depth of the luminance feature is increased.

The depth separable convolution layer separates the convolution operation and the depth changing operation through the cascaded depth convolution layer and the point-by-point convolution layer, so that the number of convolution parameters in the convolution process is smaller than that of convolution parameters of standard convolution, and the calculated amount is reduced. For example, in the process of convolving a 12×12×1 luminance image with 10 convolution kernels of 5×5×1 size and 1 step size, an 8×8×10 image feature is obtained. When the convolution is performed by the standard convolution layer, the convolution kernel is moved 8×8 times on the luminance image, and the calculated amount is 10×5×1 (8×8) =16000. When the convolution is performed by the depth separable convolution layer, the convolution is performed by 1 convolution check of 12×12×1 by 1 convolution of 5×5×1 in the depth convolution layer, so as to obtain an intermediate brightness characteristic of 8×8×1, and the calculated amount is 1×5×5×1 (8×8×1) =1600. Then convolving the intermediate luminance characteristics of 8 x 1 with 10 1 x 1 convolution checks at a point-by-point convolution layer, an image feature of 8×8×10 is obtained, and the calculated amount is 10×1×1 (1×1) ×8×1) =640. Therefore, the calculated amount of the depth-separable convolution layer is 2240 in total, and compared with the calculated amount of the normal convolution layer 16000, the calculated amount of the depth-separable convolution layer can be greatly reduced, and the processing speed can be improved.

303. The terminal maps the luminance feature to a luminance enhancement parameter of the luminance image based on the enhancement parameter perception model.

In the embodiment of the application, after the terminal determines the brightness characteristics of the brightness image through the enhancement parameter perception model, the brightness characteristics are mapped into the brightness enhancement parameters in the brightness enhancement function. The brightness enhancement function is used for determining brightness values of the pixel points in the brightness image after the pixel points are enhanced according to the brightness enhancement parameters. The luminance enhancement parameter can indicate a luminance enhancement level of a pixel point in the luminance image. For example, the larger the brightness enhancement parameter is, the larger the brightness enhancement strength of the pixel point is, and the larger the brightness value of the pixel point after enhancement is determined by the brightness enhancement function is; the smaller the brightness enhancement parameter is, the smaller the brightness enhancement strength of the pixel point is, and the smaller the brightness value of the pixel point after being enhanced is determined by the brightness enhancement function. Therefore, through the enhancement parameter perception model, different enhancement parameters can be determined according to the brightness characteristics of the brightness images of different video frames, namely different brightness enhancement forces are determined for the brightness images of different video frames.

In some embodiments, the enhanced parameter perception model further comprises a fully connected layer and an activation function layer. And the terminal processes the brightness characteristics based on the full connection layer in the enhanced parameter perception model to obtain a one-dimensional characteristic vector. Since each luminance image corresponds to one enhancement parameter, the number of output channels of the full connection layer is 1. The terminal then maps the one-dimensional feature vector to a luminance enhancement parameter based on the activation function layer. For example, the terminal uses a Tanh (hyperbolic tangent) activation function to perform nonlinear mapping on the one-dimensional feature vector, so as to obtain a brightness enhancement parameter. The full-connection layer can convert two-dimensional brightness characteristics into one-dimensional characteristic vectors, so that the influence of characteristic positions in the brightness characteristics on the output result of the enhanced parameter perception model is reduced, and the robustness of the enhanced parameter perception model is improved. And then, the one-dimensional feature vector is subjected to nonlinear mapping through the activation function layer, so that the output of the enhanced parameter perception model approaches to any nonlinear function, and the expression capacity of the whole enhanced parameter perception model is improved.

In order to more clearly explain the process of processing the input luminance image by the enhanced parameter perception model to obtain the luminance enhancement parameter of the luminance image, the above process is described below with reference to the model structure diagram of the enhanced parameter perception model shown in fig. 4. As shown in fig. 4, the enhanced parameter perception model comprises four depth separable convolutional layers, each of which comprises a cascaded depth convolutional layer and a point-by-point convolutional layer. Wherein the convolution parameters of the four depth separable convolution layers are the same. Each depth separable convolution layer is connected to one activation function layer. Illustratively, the activation function in the activation function layer connected to the depth separable convolution layer is a ReLU (Rectified Linear Unit, linear correction unit) activation function. The enhanced parameter perception model also comprises a global average pooling layer, a full connection layer, an activation function layer and a clipping layer.

The terminal inputs a brightness image to be processed into an enhanced parameter perception model, and the terminal convolves the brightness values of pixel points in the brightness image through a convolution kernel with the size of 3*3 and the step length of 2 in a depth convolution layer of a depth separable convolution layer 1 in the enhanced parameter perception model to obtain a first intermediate brightness characteristic. The terminal then convolves the first intermediate luminance feature in a point-by-point convolution layer with 16 convolution kernels of size 1*1, step size 1, to obtain a first luminance feature of depth 16. Thus, the number of input channels of the depth separable convolutional layer 1 is 1, and the number of output channels is 16. The terminal then non-linearly maps the luminance feature at depth 16 by activating the function layer. Because the convolution parameters in the four depth separable convolution layers are the same, the processing procedures in the depth separable convolution layers 2, 3 and 4 are the same as the processing procedure in the depth separable convolution layer 1, and the depth separable convolution layer 2 processes the first brightness characteristic with the depth of 16 after nonlinear mapping to obtain the second brightness characteristic with the depth of 16 until the depth separable convolution layer 4 outputs the brightness characteristic of the brightness image with the depth of 16. Thus, the number of input channels and the number of output channels of the depth-separable convolutional layer 2, the depth-separable convolutional layer 3, and the depth-separable convolutional layer 4 are each 16. The terminal then non-linearly maps the luminance characteristics of the luminance image by means of an activation function layer connected to the depth separable convolutional layer 4. And the terminal carries out global average pooling on the brightness characteristics after nonlinear mapping through a global average pooling layer, so that the size of the brightness characteristics can be reduced. And then the terminal processes the brightness characteristics through the full connection layer to obtain a one-dimensional characteristic vector. Since each luminance image corresponds to one enhancement parameter, the number of output channels of the full connection layer is 1. The terminal then maps the one-dimensional feature vector to a luminance enhancement parameter in the range of-1 to 1 based on the activation function layer. Illustratively, the activation function in the activation function layer is a Tanh (hyperbolic tangent) activation function. The terminal then clips the luminance enhancement parameters in the range of-1 to the range of 0 to 1 by a clipping function in the clipping layer. For example, the terminal clips the luminance enhancement function in the range of-1 to 0 to the luminance enhancement function in the range of 0,0 to 1, by the Clip clipping function, while the luminance enhancement function remains unchanged. Therefore, the range of the brightness enhancement parameters output by the enhancement parameter perception model is [0,1].

304. And the terminal determines the brightness value of the pixel point in the brightness image after the enhancement by a brightness enhancement function based on the brightness enhancement parameter.

In the embodiment of the application, after the terminal determines the brightness enhancement parameters in the brightness enhancement function, the terminal processes the current brightness value of the pixel point in the brightness image and the brightness threshold value of the pixel point through the brightness enhancement function and the brightness enhancement parameters in the brightness enhancement function to obtain the brightness value after the pixel point is enhanced.

First, the terminal determines a luminance threshold of the pixel point. For any pixel point in the brightness image, the terminal can determine the brightness threshold corresponding to the current brightness value of the pixel point in a locally stored brightness lookup table. The terminal can also obtain the brightness lookup table from other terminals, and then determine the brightness threshold of the pixel point according to the brightness lookup table. The brightness lookup table can indicate the corresponding relation between the brightness value and the brightness threshold value of the pixel point. The luminance threshold value is used to indicate an upper limit value of the luminance value of the pixel point. The luminance threshold is related to the current luminance value of the pixel. For example, the current luminance value of the pixel is Y, and the luminance threshold of the pixel is Y ^b . Wherein b may be a predetermined value, such as 0.5, 0.6 or 0.8, which is not limited in this embodiment of the present application. As shown in fig. 5, in the brightness enhancement curve, when the current brightness value Y of the pixel is fixed, the smaller the value of b is, the brightness threshold Y of the pixel is ^b The larger. Taking the brightness threshold value of the pixel point as Y ^b B is 0.6, and a process of determining the brightness threshold of the pixel point by the terminal through the brightness lookup table is described. The terminal determines the brightness threshold Y corresponding to the current brightness value Y of the pixel point through the brightness lookup table shown in the following table 1 ^b . The brightness threshold value of the pixel point is determined in a table look-up mode, the brightness threshold value of the pixel point is not required to be calculated through power operation, and the calculated amount is reduced. For example, in the case that the current brightness value of the pixel point is 5/255, the terminal determines that the brightness threshold value of the pixel point is 0.0945 through the brightness lookup table. Wherein, the current brightness value Y of the pixel point and the brightness threshold value Y of the pixel point in the table 1 ^b All are values after normalization.

TABLE 1

Y

0/255

1/255

2/255

3/255

4/255

5/255

…

255/255

Y ^b

0

0.0359

0.0545

0.0695

0.0826

0.0945

…

1

Then, the terminal determines a first coefficient and a second coefficient in the luminance enhancement function by the luminance enhancement parameter. The first coefficient is a coefficient of a luminance threshold of the pixel point, and the first coefficient is positively correlated with the luminance enhancement parameter. The second coefficient is the coefficient of the current brightness value of the pixel point. The second coefficient is inversely related to the luminance enhancement parameter. The terminal takes the product of the first coefficient and the brightness threshold value of the pixel point as a first brightness value and takes the product of the second coefficient and the current brightness value of the pixel point as a second brightness value. And the terminal takes the sum of the first brightness value and the second brightness value as the brightness value after the pixel point is enhanced.

For example, when the luminance enhancement parameter is 0.8, the current luminance value of the pixel is 5/255, and the luminance threshold of the pixel is 0.0945, the terminal may determine that the first coefficient is the luminance enhancement parameter 0.8, and the second coefficient is 1-0.8=0.2. The terminal determines that the first luminance value is 0.8x0.0945=0.0756, the second luminance value is 0.2x5/255=0.0039, and the luminance value after pixel enhancement is 0.0756+0.0039= 0.0795.

305. The terminal generates an intermediate brightness image based on the brightness value of the enhanced pixel point in the brightness image.

In the embodiment of the application, after the terminal determines the brightness value of the brightness image after the enhancement of a plurality of pixel points, the terminal adjusts the pixel value of the pixel point in the brightness image to the brightness value of the brightness image after the enhancement of the pixel point, so as to obtain an intermediate brightness image.

In some embodiments, the terminal normalizes luminance values of a plurality of pixels in the luminance image before the terminal extracts luminance features of the luminance image by enhancing the parametric perceptual model. For example, the terminal divides the pixel value of the pixel point by 255, and can normalize the pixel value of the pixel point to be in the range of 0-1. Correspondingly, after the terminal generates the intermediate brightness image, the brightness value of the pixel point in the intermediate brightness image is also a normalized value. Therefore, the terminal performs inverse normalization on the luminance values of the plurality of pixels in the intermediate luminance image to obtain integer luminance values of the plurality of pixels. For example, the terminal multiplies the luminance values of a plurality of pixels in the intermediate luminance image by 255 to obtain a product result, and rounds the product result to obtain an integer luminance value of the pixel. For example, for a pixel in the intermediate luminance image with a pixel value of 0.0795, the terminal determines that the enhanced integer luminance value of the pixel is 0.0975×255≡20. By normalizing the brightness values of the pixels in the brightness image before the brightness image is input into the enhanced parameter perception model, the enhanced parameter perception model can process the brightness values of the floating point number type in the brightness image, and the enhanced parameter perception model can be conveniently learned and trained. And after the intermediate brightness image is generated, the brightness values of the pixels in the intermediate brightness image are inversely normalized, so that the brightness values of the pixels in the intermediate brightness image can be restored.

306. The terminal fuses the intermediate brightness image, the image of the U channel of the first video frame in the YUV space and the image of the V channel of the first video frame in the YUV space to obtain a second video frame, wherein the second video frame is an RGB image.

In the embodiment of the application, after the terminal generates the intermediate brightness image, the terminal acquires the image of the U channel of the first video frame in the YUV space and the image of the V channel of the first video frame in the YUV space. The terminal fuses the intermediate brightness image, the image of the U channel of the first video frame in the YUV space and the image of the V channel of the first video frame in the YUV space to obtain the intermediate video frame in the YUV space. The intermediate video frame is obtained after brightness enhancement of the first video frame in YUV space. Then, the terminal converts the intermediate video frame from YUV space to RGB space to obtain a second video frame, and the brightness of the second video frame is larger than that of the first video frame.

In some embodiments, the terminal determines a luminance value of a pixel in the intermediate luminance image, a pixel value of a pixel in the image of the U-channel and a pixel value of a pixel in the image of the V-channel in the first video frame. And then, the terminal respectively processes the three pixel values through a conversion formula from YUV space to RGB space to obtain an R value, a G value and a B value after the pixel lighting brightness in the first video frame is enhanced. And the terminal determines the pixel value after the pixel brightness is enhanced according to the R value, the G value and the B value after the pixel brightness is enhanced. And the terminal adjusts the pixel value of the pixel point in the first video frame to be the pixel value after the brightness is enhanced, so as to obtain a second video frame.

In some embodiments, the terminal may first perform inter-frame smoothing on the luminance enhancement parameter to obtain a smoothed luminance enhancement parameter, and then enhance the luminance value of the pixel point in the luminance image by using the smoothed luminance enhancement parameter to obtain the second video frame.

In the first mode, the terminal performs exponential moving average on the brightness enhancement parameters of the brightness image of the first video frame based on the smooth brightness enhancement parameters of the third video frame to obtain the smooth brightness enhancement parameters of the first video frame. The third video frame is a video frame preceding the first video frame. The smooth luminance enhancement parameter of the first video frame in the video is the luminance enhancement parameter of the first video frame. And the terminal enhances the brightness value of the pixel point in the brightness image of the first video frame through the brightness enhancement function based on the smooth brightness enhancement parameter to obtain a second video frame. The brightness enhancement parameters of the first video frame are smoothed by the smoothed brightness enhancement parameters of the video frame before the first video frame, so that the inter-frame smoothing of the brightness enhancement parameters of the first video frame can be realized. Flicker phenomenon caused by discontinuous enhancement effect between adjacent video frames when the brightness of the video frames is enhanced by the brightness enhancement parameters is avoided.

For example, the terminal performs inter-frame smoothing on the luminance enhancement parameter (floating point number ranging from 0 to 1) output by the enhancement parameter perception model through an exponential moving average formula shown in the following formula (1), to obtain a smoothed enhancement parameter ranging from 0 to 1.

s _t ＝a*s _t-1 +(1-a)*x (1)

Where x is the luminance enhancement parameter of the first video frame. s is(s) _t And the smooth enhancement parameters are the smooth enhancement parameters of the first video frame, namely the smooth enhancement parameters corresponding to the brightness enhancement parameters x of the first video frame output by the enhancement parameter perception model at the moment t. s is(s) _t-1 And the smooth enhancement parameters of the third video frame are the smooth enhancement parameters corresponding to the brightness enhancement parameters of the third video frame output by the enhancement parameter perception model at the time t-1. a is a preset weight, and the value range of a is 0 to 1. The larger the value of a, the stronger the smoothing effect, the smaller the value of a, and the weaker the smoothing effect. Illustratively, when a is 0.5, the luminance enhancement parameter can be suitably smoothed. Smooth luminance enhancement parameter s for the first video frame in a video ₀ Is the luminance enhancement parameter for the first video frame.

And in a second mode, the terminal carries out exponential moving average on the brightness enhancement parameters of the brightness images of the first video frame based on the brightness enhancement parameters of the N fourth video frames to obtain smooth brightness enhancement parameters of the first video frame. N is a positive integer, and the frame number of the fourth video frame is smaller than that of the first video frame. And the terminal enhances the brightness value of the pixel point in the brightness image of the first video frame through the brightness enhancement function based on the smooth brightness enhancement parameter to obtain a second video frame.

When the terminal performs exponential moving average on the brightness enhancement parameters of the brightness images of the first video frames according to the brightness enhancement parameters of the N fourth video frames, the weight of the brightness enhancement parameters of the fourth video frames is inversely related to the difference between the frame numbers of the fourth video frames and the frame numbers of the first video frames. For example, in the case where N is 3 and the first video frame is video frame No. 4, the terminal acquires luminance enhancement parameters of the first 3 video frames (video frame No. 1, video frame No. 2, and video frame No. 3) of the first video frame. The terminal determines that the weight of the brightness enhancement parameter x of the first video frame is 0.4, the weight of the brightness enhancement parameter of the No. 3 video frame is 0.3, the weight of the brightness enhancement parameter of the No. 2 video frame is 0.2, and the weight of the brightness enhancement parameter of the No. 1 video frame is 0.1. By giving higher weight to the luminance enhancement parameters of the video frame closer to the first video frame at the time of exponential moving average, it is possible to refer more to the enhancement parameters of the previous video frame when determining the smooth enhancement parameters of the first video frame. Therefore, the brightness enhancement strength between adjacent video frames is smoother, and the enhancement effect between the video frames is ensured to be continuous.

Both of the above methods smooth the luminance enhancement parameter by exponential moving average. In some embodiments, the terminal may also perform inter-frame smoothing on the brightness enhancement parameters by other smoothing methods, such as ridge regression, which is not limited in this embodiment of the present application.

In some embodiments, the terminal processes the current brightness value of the pixel point and the brightness threshold value of the pixel point in the brightness image through the brightness enhancement function and the smooth enhancement parameter to obtain the brightness value after the pixel point is enhanced. The current brightness value of the pixel point is taken as Y, and the brightness threshold value of the pixel point is taken as Y ^b B is 0.6 as an example. In order to reduce the brightness threshold Y obtained by performing power operation on the current brightness value Y of the pixel point by the terminal ^b Is calculated by the terminalThe brightness threshold Y corresponding to the current brightness value Y of the pixel point can be directly determined by searching the brightness lookup table ^b . For any pixel point in the brightness image, the terminal determines the brightness threshold corresponding to the current brightness value of the pixel point in the brightness lookup table shown in the table 1. After the terminal determines the brightness threshold value of the pixel point, the brightness threshold value of the pixel point and the current brightness value of the pixel point are processed through a brightness enhancement function based on the smooth brightness enhancement parameter, so that the brightness value of the pixel point after the enhancement is obtained. The terminal generates a second video frame based on the enhanced luminance values of the plurality of pixels. The brightness threshold value of the pixel point is determined by searching the brightness lookup table, the current brightness value of the pixel point in the brightness image is not required to be subjected to power operation to obtain the brightness threshold value of the pixel point, the calculated amount is reduced, and the processing speed is improved.

For example, the terminal determines the luminance value after the pixel point enhancement by a luminance enhancement function shown in the following formula (2).

Y _new ＝s _t *Y ^b +(1-s _t )*Y (2)

Wherein Y is the current luminance value of the pixel point in the luminance image of the first video frame. Y is Y ^b Is the luminance threshold of the pixel point in the luminance image of the first video frame. s is(s) _t Is a smooth enhancement parameter for the first video frame. Y is Y _new And the brightness value is the brightness value of the enhanced pixel point in the brightness image of the first video frame.

In some embodiments, the terminal may determine, in the luminance enhancement function, a current luminance value of the pixel and a coefficient corresponding to a luminance threshold of the pixel, and determine, by using the coefficient, the luminance value of the pixel after enhancement. The terminal determines a first coefficient and a second coefficient in the luminance enhancement function based on the smoothed luminance enhancement parameter. The first coefficient is a coefficient of a brightness threshold of the pixel point, and the first coefficient is positively correlated with the smooth brightness enhancement parameter. The second coefficient is the coefficient of the current brightness value of the pixel point, and the second coefficient is inversely related to the smooth brightness enhancement parameter. And the terminal respectively processes the brightness threshold value of the pixel point and the current brightness value of the pixel point based on the first coefficient and the second coefficient in the brightness enhancement function to obtain a first brightness value and a second brightness value. The first brightness value is the product of the first coefficient and the brightness threshold value of the pixel point, and the second brightness value is the product of the second coefficient and the current brightness value of the pixel point. And the terminal takes the sum of the first brightness value and the second brightness value as the brightness value after the pixel point is enhanced.

For example, the terminal determines the first coefficient as the smoothing enhancement parameter s _t The second coefficient is (1-s _t ). The first brightness value is s _t *Y ^b . The second brightness value is (1-s) _t ) Y. The brightness value after the pixel point is enhanced is s _t *Y ^b +(1-s _t )*Y。

In order to more clearly explain the process of enhancing the luminance of the first video frame by the terminal, the above process will be described with reference to a flowchart of luminance enhancement shown in fig. 6. As shown in fig. 6, the terminal converts the first video frame with brightness to be enhanced from RGB space to YUV space, to obtain a first video frame in YUV format. And then the terminal separates brightness channels in the first video frame in YUV format to obtain brightness image of the first video frame. The terminal divides the brightness value of the pixel point in the brightness image by 255 to normalize. And the terminal downsamples the normalized brightness image, and reduces the resolution of the brightness image. And the terminal processes the brightness image after downsampling through the enhancement parameter perception model to obtain brightness enhancement parameters of the first video frame output by the enhancement parameter perception model. And the terminal performs inter-frame smoothing on the brightness enhancement parameters of the first video frame by adopting an exponential moving average method to obtain smooth enhancement parameters. And then the terminal enhances the brightness value of each pixel point in the brightness image by smoothing the enhancement parameters to obtain an intermediate brightness image. And the terminal performs inverse normalization on the brightness value of the pixel point in the intermediate brightness image to obtain an integer brightness value. And then the terminal fuses the intermediate brightness image, the image corresponding to the first video frame in the U channel and the image corresponding to the first video frame in the V channel, and converts the fused image from YUV space to RGB space to obtain a second video frame.

Therefore, the terminal can carry out brightness enhancement on the video frame with lower brightness through the processing flow, and the brightness of the video frame is improved. Even under the condition of poor light of the video recording environment, the brightness of the video picture can be improved, so that the video picture is clearer. As shown in the schematic diagram of the video frame in fig. 7, the left side of fig. 7 is the video frame to be enhanced in brightness, and the right side of fig. 7 is the video frame after brightness enhancement. As can be seen from fig. 7, the brightness of the right video frame is higher and the picture is clearer. In the process that the terminal enhances the brightness of the video frame, the terminal can adaptively determine the brightness enhancement parameter of the video frame through the enhancement parameter perception model. That is, the terminal is able to determine different brightness enhancement levels for video frames of different brightness. Compared with a mode of reinforcing all video frames by adopting fixed brightness enhancement force, the method avoids the problem that a part of originally brighter video frames generate white pictures after brightness enhancement. As shown in the schematic diagram of the video frame in fig. 8, the left side of fig. 8 is a video frame obtained by enhancing the video frame to be processed with a fixed brightness enhancement force. The right side of fig. 8 is a video frame obtained by enhancing a video frame to be processed by the luminance enhancement method provided by the embodiment of the present application. As can be seen from fig. 8, the left video frame is whitened, the contrast is low, and the enhancement effect is more general. The right side video frame does not generate the problem of whitening of the picture, and the enhancement effect is higher than that of the left side video frame.

The above embodiments are described taking the luminance of a video frame in a terminal enhanced video as an example. In some embodiments, the terminal may also send at least one video frame to be enhanced in brightness to the server, which enhances the brightness of the video frame. And the server sends the video frame with enhanced brightness to the terminal, and the terminal displays the brightness enhancement result of the video frame.

On the basis of the embodiment, in order to ensure the accuracy of the enhanced parameter sensing model, the enhanced parameter sensing model needs to be trained. The training process for enhancing the parametric perceptual model is described in the following examples.

Fig. 9 is a flowchart of a training method for enhancing a parameter sensing model according to an embodiment of the present application, and as shown in fig. 9, the training method is described by using a server as an example in the embodiment of the present application. The training method of the enhanced parameter perception model comprises the following steps:

901. the server acquires a brightness image of the sample image and a label image of the sample image, wherein the label image is an image obtained by enhancing the brightness of the sample image.

In an embodiment of the present application, the sample image includes an original image and a darker image obtained by performing brightness reduction of different intensities on the original image. The same procedure for obtaining the luminance image of the sample image by the server as that for obtaining the luminance image of the first video frame in step 301 is not described here again. In some embodiments, the label image of the sample image is an image obtained by manually adjusting the brightness of the original image to a brightness that is comfortable for the human eye.

902. The server extracts brightness features of the brightness image based on an enhancement parameter perception model, maps the brightness features to brightness enhancement parameters in a brightness enhancement function, wherein the enhancement parameter perception model is used for determining brightness enhancement parameters of the input image in the brightness enhancement function based on brightness values of a plurality of pixel points in the input image, and the brightness enhancement function is used for determining brightness values of the pixel points after the pixel points are enhanced according to the brightness enhancement parameters.

In the embodiment of the application, the server convolves the brightness values of the pixel points in the brightness image of the sample image through the enhanced parameter perception model to obtain the brightness characteristics of the brightness image. The server maps the luminance characteristics into luminance enhancement parameters of the luminance image in a luminance enhancement function through an enhancement parameter perception model. The specific process of determining the brightness enhancement parameters is the same as that of step 302 and step 303, and is not described herein.

903. And the server enhances the brightness value of the pixel point in the brightness image through a brightness enhancement function based on the brightness enhancement parameter, and obtains a sample image after brightness enhancement.

In the embodiment of the application, the server processes the current brightness value of the pixel point in the brightness image and the brightness threshold value of the pixel point through the brightness enhancement function and the brightness enhancement parameter in the brightness enhancement function to obtain the brightness value after the pixel point is enhanced. Wherein the service is able to determine the luminance threshold of the pixel point by looking up a luminance look-up table. The luminance lookup table can indicate a correspondence between luminance values and luminance threshold values of the pixel points. And the server adjusts the pixel value of the pixel point in the brightness image to the enhanced brightness value to obtain an intermediate brightness image. And then the server generates a sample image with enhanced brightness according to the intermediate brightness image and the image of the V channel of the image of the U channel of the sample image in the YUV space.

904. The server determines a training loss of the enhanced parameter perception model based on the sample image and the label image after brightness enhancement.

In the embodiment of the application, the server takes the difference between the sample image and the label image after brightness enhancement as the training loss of the enhanced parameter perception model.

In some embodiments, the server is able to determine the training loss of the enhanced parameter-aware model by a loss function shown in equation (3) below.

L1＝|I _gt -E(I,f(I))| (3)

Wherein L1 is an L1 penalty function. I _gt For label images. E is a sample image subjected to luminance enhancement by the luminance enhancement method provided by the above embodiment. I is a sample image. f is an enhanced parameter perception model.

It should be noted that the server can determine the training loss through other loss functions, such as the L2 loss function, the GAN loss function, and the perceptual loss function, which is not limited in this embodiment of the present application.

905. The server trains the enhanced parameter perception model based on the training loss.

In the embodiment of the application, the server adopts an error back propagation algorithm to train the enhanced parameter perception model. The server updates model parameters of the enhanced parameter perception model based on the training loss, so that the loss value of the enhanced parameter perception model is reduced, and the updated enhanced parameter perception model is obtained through training. And if the updated enhancement parameter perception model meets the training ending condition, if the training times are the target times or the training loss of the enhancement parameter perception model is in the target range, taking the updated enhancement parameter perception model as the enhancement parameter perception model after training. If the updated enhancement parameter sensing model does not meet the training ending condition, updating the enhancement parameter sensing model again according to the modes from step 902 to step 904 until the updated enhancement parameter sensing model meets the training ending condition, and obtaining the enhancement parameter sensing model after training is completed.

The embodiment of the application provides a training method of an enhanced parameter perception model, and a server processes brightness values of pixel points in a brightness image of a sample image based on the enhanced parameter perception model to obtain brightness characteristics of the brightness image. The server then maps the luminance characteristics to luminance enhancement parameters of the luminance image via an enhancement parameter perception model. The server enhances the brightness of the brightness image through the brightness enhancement parameters to obtain a sample image with enhanced brightness. In the process of training the enhancement parameter perception model, the server can determine the training loss of the enhancement parameter perception model based on the label image of the sample image and the sample image after brightness enhancement, and train the enhancement parameter perception model through the training loss, so that the enhancement parameter perception model learns the capability of extracting brightness characteristics and the capability of adaptively perceiving brightness enhancement strength according to the brightness characteristics, and the accuracy of the enhancement parameter perception model is ensured.

Fig. 10 is a schematic structural diagram of a brightness enhancement device according to an embodiment of the present application. Referring to fig. 10, the apparatus includes: an acquisition module 1001, a determination module 1002, and a brightness enhancement module 1003.

An obtaining module 1001, configured to obtain a luminance image of a first video frame in a video, where the luminance image is an image of a luminance channel of the first video frame in a YUV space;

a determining module 1002, configured to extract luminance characteristics of a luminance image based on an enhancement parameter sensing model, map the luminance characteristics to luminance enhancement parameters in a luminance enhancement function, and determine luminance enhancement parameters of the input image in the luminance enhancement function based on luminance values of a plurality of pixel points in the input image, where the luminance enhancement function is configured to determine luminance values of the pixel points after enhancement according to the luminance enhancement parameters;

the luminance enhancement module 1003 is configured to enhance luminance values of pixels in the luminance image by a luminance enhancement function based on the luminance enhancement parameter, so as to obtain a second video frame, where luminance of the second video frame is greater than that of the first video frame.

In some embodiments, the luminance enhancement module 1003 is configured to exponentially moving average luminance enhancement parameters of a luminance image of a first video frame based on a smooth luminance enhancement parameter of a third video frame, to obtain a smooth luminance enhancement parameter of the first video frame, where the third video frame is a previous video frame of the first video frame, and the smooth luminance enhancement parameter of the first video frame in the video is the luminance enhancement parameter of the first video frame; and on the basis of the smooth brightness enhancement parameters, enhancing the brightness values of pixel points in the brightness image of the first video frame through a brightness enhancement function to obtain a second video frame.

In some embodiments, the luminance enhancement module 1003 is configured to exponentially moving average luminance enhancement parameters of a luminance image of a first video frame based on luminance enhancement parameters of N fourth video frames, where N is a positive integer, and a frame number of the fourth video frame is smaller than a frame number of the first video frame; and on the basis of the smooth brightness enhancement parameters, enhancing the brightness values of pixel points in the brightness image of the first video frame through a brightness enhancement function to obtain a second video frame.

In some embodiments, fig. 11 is a schematic structural diagram of another brightness enhancement device according to an embodiment of the present application, as shown in fig. 11, a brightness enhancement module 1003, including:

a determining unit 10031, configured to determine, for any pixel in the luminance image, a luminance threshold corresponding to a current luminance value of the pixel in a luminance lookup table, where the luminance lookup table is used to indicate a correspondence between the luminance value of the pixel and the luminance threshold, and the luminance threshold is used to indicate an upper limit value of the luminance value of the pixel;

the brightness enhancement unit 10032 is configured to process, based on the smooth brightness enhancement parameter, the brightness threshold value of the pixel point and the current brightness value of the pixel point by using a brightness enhancement function, so as to obtain a brightness value after the pixel point is enhanced;

The generating unit 10033 is configured to generate a second video frame based on the luminance values after the plurality of pixel points are enhanced.

In some embodiments, the luminance enhancement unit 10032 is configured to determine, based on the smoothed luminance enhancement parameter, a first coefficient and a second coefficient in the luminance enhancement function, where the first coefficient is a coefficient of a luminance threshold of the pixel, the first coefficient is positively related to the smoothed luminance enhancement parameter, the second coefficient is a coefficient of a current luminance value of the pixel, and the second coefficient is negatively related to the smoothed luminance enhancement parameter; processing a brightness threshold value of the pixel point and a current brightness value of the pixel point respectively based on a first coefficient and a second coefficient in the brightness enhancement function to obtain a first brightness value and a second brightness value, wherein the first brightness value is the product of the first coefficient and the brightness threshold value of the pixel point, and the second brightness value is the product of the second coefficient and the current brightness value of the pixel point; and taking the sum of the first brightness value and the second brightness value as the brightness value after the pixel point is enhanced.

a determining module 1002, configured to convolve luminance values of a plurality of pixel points in a luminance image based on a depth convolution layer in the depth separable convolution layers, to obtain an intermediate luminance feature of the luminance image, where a depth of the intermediate luminance feature is 1; based on the point-by-point convolution layer in the depth separable convolution layer, the middle brightness characteristic is convolved to obtain the brightness characteristic of the brightness image, wherein the depth of the brightness characteristic is the same as the number of output channels of the convolution kernel in the point-by-point convolution layer.

a determining module 1002, configured to process the luminance feature based on the full connection layer to obtain a one-dimensional feature vector; the one-dimensional feature vector is mapped to a luminance enhancement parameter based on the activation function layer.

In some embodiments, the luminance enhancement module 1003 is configured to determine, based on the luminance enhancement parameter, a luminance value of the enhanced pixel point in the luminance image by using a luminance enhancement function; generating an intermediate brightness image based on the brightness value of the enhanced pixel point in the brightness image; and fusing the intermediate brightness image, the image of the U channel of the first video frame in the YUV space and the image of the V channel of the first video frame in the YUV space to obtain a second video frame, wherein the second video frame is an RGB image.

In some embodiments, the apparatus further comprises:

a normalization module 1004, configured to normalize luminance values of a plurality of pixels in the luminance image;

the inverse normalization module 1005 is configured to inversely normalize luminance values of a plurality of pixels in the intermediate luminance image to obtain integer luminance values of the plurality of pixels.

In some embodiments, the apparatus further comprises:

a downsampling module 1006, configured to downsample the luminance image.

The embodiment of the application provides a brightness enhancement device. In the process of enhancing the brightness of a video frame in a video, a brightness image of the video frame in a brightness channel is obtained. And extracting the brightness characteristics of the brightness image according to the current brightness value of each pixel point in the brightness image by the enhanced parameter perception model. The enhancement parameter perception model adaptively determines a luminance enhancement parameter of a luminance image in a luminance enhancement function according to a luminance characteristic of the luminance image. The brightness enhancement function can determine the brightness value of the pixel after the pixel is enhanced according to the current brightness value of the pixel, and the brightness enhancement parameter is used for determining the enhancement strength of the brightness value of the pixel. Therefore, the enhancement parameter perception model can adaptively perceive the enhancement parameters of different video frames, namely the brightness enhancement strength of different video frames according to the brightness characteristics of different video frames. Compared with the mode of reinforcing all video frames by adopting fixed reinforcing force, the processing mode avoids the problem that part of video frames are too dark or too bright, and improves the brightness reinforcing effect of the video frames.

It should be noted that: the brightness enhancement device provided in the above embodiment is only exemplified by the division of the above functional modules, and in practical application, the above functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the terminal is divided into different functional modules, so as to perform all or part of the functions described above. In addition, the embodiments of the brightness enhancement device and the brightness enhancement method provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the embodiments of the method are detailed in the method embodiments, which are not repeated herein.

Fig. 12 is a schematic structural diagram of an enhanced parameter perception model training device according to an embodiment of the present application. The apparatus is used for executing the enhanced parameter perception model training method, referring to fig. 12, the apparatus includes: an acquisition module 1201, a determination module 1202, a brightness enhancement module 1203, a training loss determination module 1204, and a training module 1205.

An obtaining module 1201, configured to obtain a brightness image of a sample image and a label image of the sample image, where the label image is an image obtained by enhancing the brightness of the sample image;

a determining module 1202, configured to extract a luminance feature of a luminance image based on an enhancement parameter perception model, map the luminance feature to a luminance enhancement parameter in a luminance enhancement function, and determine the luminance enhancement parameter of the input image in the luminance enhancement function based on luminance values of a plurality of pixels in the input image, where the luminance enhancement function is configured to determine the luminance value of the enhanced pixels according to the luminance enhancement parameter;

the brightness enhancement module 1203 is configured to enhance, based on the brightness enhancement parameter, a brightness value of a pixel point in the brightness image by using a brightness enhancement function, so as to obtain a sample image after brightness enhancement;

a training loss determination module 1204, configured to determine a training loss of the enhanced parameter perception model based on the sample image and the label image after brightness enhancement;

A training module 1205 for training the enhanced parameter perception model based on the training loss.

The embodiment of the application provides an enhanced parameter perception model training device, which is used for processing the brightness value of a pixel point in a brightness image of a sample image based on an enhanced parameter perception model by a server to obtain the brightness characteristic of the brightness image. The server then maps the luminance characteristics to luminance enhancement parameters of the luminance image via an enhancement parameter perception model. The server enhances the brightness of the brightness image through the brightness enhancement parameters to obtain a sample image with enhanced brightness. In the process of training the enhancement parameter perception model, the server can determine the training loss of the enhancement parameter perception model based on the label image of the sample image and the sample image after brightness enhancement, and train the enhancement parameter perception model through the training loss, so that the enhancement parameter perception model learns the capability of extracting brightness characteristics and the capability of adaptively perceiving brightness enhancement strength according to the brightness characteristics, and the accuracy of the enhancement parameter perception model is ensured.

It should be noted that: the enhanced parameter perception model training device provided in the above embodiment is only exemplified by the division of the above functional modules, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the enhanced parameter perception model training device and the enhanced parameter perception model training method provided in the foregoing embodiments belong to the same concept, and detailed implementation processes of the enhanced parameter perception model training device and the enhanced parameter perception model training method are detailed in the method embodiments, which are not described herein.

In the embodiment of the present application, the computer device can be configured as a terminal or a server, when the computer device is configured as a terminal, the technical solution provided by the embodiment of the present application may be implemented by the terminal as an execution body, and when the computer device is configured as a server, the technical solution provided by the embodiment of the present application may be implemented by the server as an execution body, or the technical solution provided by the present application may be implemented by interaction between the terminal and the server, which is not limited by the embodiment of the present application.

Fig. 13 is a schematic structural diagram of a terminal according to an embodiment of the present application.

The terminal 1300 includes: a processor 1301, and a memory 1302.

Processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. Processor 1301 may be implemented in hardware in at least one of DSP (Digital Signal Processing ), FPGA (Field Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). Processor 1301 may also include a main processor, which is a processor for processing data in an awake state, also called a CPU (Central Processing Unit ), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, processor 1301 may integrate a GPU (Graphics Processing Unit, image enhanced interactor) for responsible for rendering and rendering of content required to be displayed by the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.

Memory 1302 may include one or more computer-readable storage media, which may be non-transitory. Memory 1302 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1302 is used to store at least one computer program for execution by processor 1301 to implement the brightness enhancement method provided by the method embodiments of the present application or to implement the training method of the enhanced parameter perception model provided by the method embodiments of the present application.

In some embodiments, the terminal 1300 may further optionally include: a peripheral interface 1303 and at least one peripheral. The processor 1301, the memory 1302, and the peripheral interface 1303 may be connected by a bus or signal lines. The respective peripheral devices may be connected to the peripheral device interface 1303 through a bus, a signal line, or a circuit board. Optionally, the peripheral device comprises: at least one of radio frequency circuitry 1304, a display screen 1305, a camera assembly 1306, audio circuitry 1307, and a power supply 1308.

A peripheral interface 1303 may be used to connect I/O (Input/Output) related at least one peripheral to the processor 1301 and the memory 1302. In some embodiments, processor 1301, memory 1302, and peripheral interface 1303 are integrated on the same chip or circuit board; in some other embodiments, either or both of the processor 1301, the memory 1302, and the peripheral interface 1303 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The Radio Frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 1304 converts an electrical signal to an electromagnetic signal for transmission, or converts a received electromagnetic signal to an electrical signal. Optionally, the radio frequency circuit 1304 includes: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and so forth. The radio frequency circuit 1304 may communicate with other devices via at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity ) networks. In some embodiments, the radio frequency circuit 1304 may also include NFC (Near Field Communication ) related circuits, which the present application is not limited to.

The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display 1305 is a touch display, the display 1305 also has the ability to capture touch signals at or above the surface of the display 1305. The touch signal may be input to the processor 1301 as a control signal for processing. At this point, the display 1305 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display screen 1305 may be one and disposed on the front panel of the terminal 1300; in other embodiments, the display 1305 may be at least two, disposed on different surfaces of the terminal 1300 or in a folded configuration; in other embodiments, the display 1305 may be a flexible display disposed on a curved surface or a folded surface of the terminal 1300. Even more, the display screen 1305 may be arranged in a non-rectangular irregular pattern, i.e., a shaped screen. The display screen 1305 may be made of LCD (Liquid Crystal Display ), OLED (Organic Light-Emitting Diode) or other materials.

The camera assembly 1306 is used to capture images or video. Optionally, camera assembly 1306 includes a front camera and a rear camera. The front camera is disposed on the front panel of the terminal 1300, and the rear camera is disposed on the rear surface of the terminal 1300. In some embodiments, the at least two rear cameras are any one of a main camera, a depth camera, a wide-angle camera and a tele camera, so as to realize that the main camera and the depth camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize a panoramic shooting and Virtual Reality (VR) shooting function or other fusion shooting functions. In some embodiments, camera assembly 1306 may also include a flash. The flash lamp can be a single-color temperature flash lamp or a double-color temperature flash lamp. The dual-color temperature flash lamp refers to a combination of a warm light flash lamp and a cold light flash lamp, and can be used for light compensation under different color temperatures.

The audio circuit 1307 may include a microphone and a speaker. The microphone is used for collecting sound waves of users and environments, converting the sound waves into electric signals, and inputting the electric signals to the processor 1301 for processing, or inputting the electric signals to the radio frequency circuit 1304 for voice communication. For purposes of stereo acquisition or noise reduction, a plurality of microphones may be provided at different portions of the terminal 1300, respectively. The microphone may also be an array microphone or an omni-directional pickup microphone. The speaker is then used to convert electrical signals from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only the electric signal can be converted into a sound wave audible to humans, but also the electric signal can be converted into a sound wave inaudible to humans for ranging and other purposes. In some embodiments, the audio circuit 1307 may also comprise a headphone jack.

A power supply 1308 is used to power the various components in terminal 1300. The power source 1308 may be alternating current, direct current, a disposable battery, or a rechargeable battery. When the power source 1308 comprises a rechargeable battery, the rechargeable battery may support wired or wireless charging. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, terminal 1300 also includes one or more sensors 1309. The one or more sensors 1309 include, but are not limited to: acceleration sensor 1310, gyroscope sensor 1311, pressure sensor 1312, optical sensor 1313, and proximity sensor 1314.

The acceleration sensor 1310 may detect the magnitudes of accelerations on three coordinate axes of the coordinate system established with the terminal 1300. For example, the acceleration sensor 1310 may be used to detect components of gravitational acceleration in three coordinate axes. Processor 1301 may control display screen 1305 to display a user interface in either a landscape view or a portrait view based on the gravitational acceleration signal acquired by acceleration sensor 1310. Acceleration sensor 1310 may also be used for the acquisition of motion data for games or users.

The gyro sensor 1311 may detect a body direction and a rotation angle of the terminal 1300, and the gyro sensor 1311 may collect a 3D motion of the user to the terminal 1300 in cooperation with the acceleration sensor 1310. Processor 1301 can implement the following functions based on data collected by gyro sensor 1311: motion sensing (e.g., changing UI according to a tilting operation by a user), image stabilization at shooting, game control, and inertial navigation.

Pressure sensor 1312 may be disposed on a side frame of terminal 1300 and/or on an underlying layer of display screen 1305. When the pressure sensor 1312 is disposed at a side frame of the terminal 1300, a grip signal of the terminal 1300 by a user may be detected, and the processor 1301 performs left-right hand recognition or shortcut operation according to the grip signal collected by the pressure sensor 1312. When the pressure sensor 1312 is disposed at the lower layer of the display screen 1305, the processor 1301 controls the operability control on the UI interface according to the pressure operation of the user on the display screen 1305. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

The optical sensor 1313 is used to collect ambient light intensity. In one embodiment, processor 1301 may control the display brightness of display screen 1305 based on the intensity of ambient light collected by optical sensor 1313. Optionally, when the ambient light intensity is high, the display brightness of the display screen 1305 is turned high; when the ambient light intensity is low, the display brightness of the display screen 1305 is turned down. In another embodiment, processor 1301 may also dynamically adjust the shooting parameters of camera assembly 1306 based on the intensity of ambient light collected by optical sensor 1313.

A proximity sensor 1314, also referred to as a distance sensor, is provided on the front panel of terminal 1300. Proximity sensor 1314 is used to capture the distance between the user and the front of terminal 1300. In one embodiment, when proximity sensor 1314 detects that the distance between the user and the front of terminal 1300 gradually decreases, processor 1301 controls display screen 1305 to switch from the bright screen state to the off screen state; when the proximity sensor 1314 detects that the distance between the user and the front surface of the terminal 1300 gradually increases, the processor 1301 controls the display screen 1305 to switch from the off-screen state to the on-screen state.

Those skilled in the art will appreciate that the structure shown in fig. 13 is not limiting of terminal 1300 and may include more or fewer components than shown, or may combine certain components, or may employ a different arrangement of components.

Fig. 14 is a schematic structural diagram of a server according to an embodiment of the present application, where the server 1400 may have a relatively large difference due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1401 and one or more memories 1402, where the memories 1402 store at least one computer program, and the at least one computer program is loaded and executed by the processor 1401 to implement the brightness enhancement methods provided by the above-mentioned method embodiments, or implement the training method of the enhanced parameter perception model provided by the above-mentioned method embodiments. Of course, the server 1400 may also have a wired or wireless network interface, a keyboard, and an input/output interface, so as to perform input/output, and the server 1400 may also include other components for implementing the functions of the device, which are not described herein.

The embodiment of the application also provides a computer readable storage medium, in which at least one section of computer program is stored, and the at least one section of computer program is loaded and executed by a processor to implement the brightness enhancement method in the above embodiment, or implement the training method for enhancing the parameter perception model in the above embodiment. For example, the computer readable storage medium may be Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), compact disk Read-Only Memory (Compact Disc Read-Only Memory, CD-ROM), magnetic tape, floppy disk, optical data storage device, etc.

Embodiments of the present application also provide a computer program product, including a computer program product, which is executed by a processor to implement the brightness enhancement method in the above embodiment, or to implement the training method for enhancing the parameter perception model in the above embodiment.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The foregoing description of the embodiments of the application is merely illustrative of the principles of the embodiments of the present application, and various modifications, equivalents, improvements, etc. may be made without departing from the spirit and principles of the embodiments of the application.

Claims

1. A method of brightness enhancement, the method comprising:

2. The method according to claim 1, wherein the enhancing the luminance value of the pixel in the luminance image by the luminance enhancement function based on the luminance enhancement parameter to obtain the second video frame includes:

performing exponential moving average on the brightness enhancement parameters of the brightness image of the first video frame based on the smooth brightness enhancement parameters of a third video frame, so as to obtain the smooth brightness enhancement parameters of the first video frame, wherein the third video frame is the previous video frame of the first video frame, and the smooth brightness enhancement parameters of the first video frame in the video are the brightness enhancement parameters of the first video frame;

and based on the smooth brightness enhancement parameters, enhancing the brightness values of pixel points in the brightness image of the first video frame through the brightness enhancement function to obtain the second video frame.

3. The method according to claim 1, wherein the enhancing the luminance value of the pixel in the luminance image by the luminance enhancement function based on the luminance enhancement parameter to obtain the second video frame includes:

performing exponential moving average on the brightness enhancement parameters of the brightness images of the first video frames based on the brightness enhancement parameters of N fourth video frames to obtain smooth brightness enhancement parameters of the first video frames, wherein N is a positive integer, and the frame number of the fourth video frames is smaller than that of the first video frames;

4. A method according to claim 2 or 3, wherein said enhancing the luminance value of the pixel in the luminance image of the first video frame by the luminance enhancement function based on the smoothed luminance enhancement parameter to obtain the second video frame comprises:

for any pixel point in the brightness image, determining a brightness threshold value corresponding to the current brightness value of the pixel point in a brightness lookup table, wherein the brightness lookup table is used for indicating the corresponding relation between the brightness value of the pixel point and the brightness threshold value, and the brightness threshold value is used for indicating the upper limit value of the brightness value of the pixel point;

processing the brightness threshold value of the pixel point and the current brightness value of the pixel point through the brightness enhancement function based on the smooth brightness enhancement parameter to obtain the brightness value of the pixel point after enhancement;

and generating the second video frame based on the brightness values of the enhanced pixel points.

5. The method of claim 4, wherein the processing, based on the smooth luminance enhancement parameter, the luminance threshold value of the pixel and the current luminance value of the pixel by the luminance enhancement function to obtain the luminance value of the pixel after enhancement comprises:

Determining a first coefficient and a second coefficient in the brightness enhancement function based on the smooth brightness enhancement parameter, wherein the first coefficient is a coefficient of a brightness threshold value of the pixel point, the first coefficient is positively related to the smooth brightness enhancement parameter, the second coefficient is a coefficient of a current brightness value of the pixel point, and the second coefficient is negatively related to the smooth brightness enhancement parameter;

based on a first coefficient and a second coefficient in the brightness enhancement function, respectively processing a brightness threshold value of the pixel point and a current brightness value of the pixel point to obtain a first brightness value and a second brightness value, wherein the first brightness value is the product of the first coefficient and the brightness threshold value of the pixel point, and the second brightness value is the product of the second coefficient and the current brightness value of the pixel point;

and taking the sum of the first brightness value and the second brightness value as the brightness value after the pixel point is enhanced.

6. The method of claim 1, wherein the enhanced parameter perception model comprises a depth separable convolutional layer comprising a depth convolutional layer and a point-by-point convolutional layer;

The extracting the brightness characteristics of the brightness image based on the enhanced parameter perception model comprises the following steps:

convolving the brightness values of a plurality of pixel points in the brightness image based on a depth convolution layer in the depth separable convolution layer to obtain an intermediate brightness characteristic of the brightness image, wherein the depth of the intermediate brightness characteristic is 1;

and convolving the intermediate brightness characteristic based on a point-by-point convolution layer in the depth separable convolution layer to obtain the brightness characteristic of the brightness image, wherein the depth of the brightness characteristic is the same as the number of output channels of a convolution kernel in the point-by-point convolution layer.

7. The method of claim 1, wherein the enhanced parameter aware model further comprises a full connectivity layer and an activation function layer;

the mapping the luminance characteristics to luminance enhancement parameters in a luminance enhancement function includes:

processing the brightness characteristics based on the full connection layer to obtain one-dimensional characteristic vectors;

the one-dimensional feature vector is mapped to the luminance enhancement parameter based on the activation function layer.

8. The method according to claim 1, wherein the enhancing the luminance value of the pixel in the luminance image by the luminance enhancement function based on the luminance enhancement parameter to obtain the second video frame includes:

Determining a brightness value of the enhanced pixel point in the brightness image through the brightness enhancement function based on the brightness enhancement parameter;

generating an intermediate brightness image based on the brightness value of the enhanced pixel point in the brightness image;

and fusing the intermediate brightness image, the image of the U channel of the first video frame in the YUV space and the image of the V channel of the first video frame in the YUV space to obtain the second video frame, wherein the second video frame is an RGB image.

9. The method of claim 8, wherein the extracting luminance features of the luminance image based on the enhancement parameter perception model, before mapping the luminance features to luminance enhancement parameters in a luminance enhancement function, further comprises:

normalizing brightness values of a plurality of pixel points in the brightness image;

after the intermediate brightness image is generated based on the brightness value of the enhanced pixel point in the brightness image, the method further comprises:

and carrying out inverse normalization on brightness values of a plurality of pixel points in the intermediate brightness image to obtain integer brightness values of the plurality of pixel points.

10. The method of claim 9, wherein the extracting luminance features of the luminance image based on the enhancement parameter perception model, before mapping the luminance features to luminance enhancement parameters in a luminance enhancement function, further comprises:

And downsampling the brightness image.

11. A method of training an enhanced parametric perceptual model, the method comprising:

based on the training loss, training the enhanced parameter perception model.

12. A brightness enhancement device, the device comprising:

13. A training device for enhancing a parametric perceptual model, the device comprising:

14. A computer device comprising a processor and a memory, wherein the memory has stored therein at least one computer program that is loaded and executed by the processor to implement the brightness enhancement method of any one of claims 1 to 10 or the training method of the enhanced parameter perception model of claim 11.

15. A computer readable storage medium, wherein at least one computer program is stored in the computer readable storage medium, the at least one computer program being loaded and executed by a processor to implement the brightness enhancement method of any one of claims 1 to 10 or the training method of the enhanced parameter perception model of claim 11.

16. A computer program product comprising a computer program, characterized in that the computer program is loaded and executed by a processor to implement a brightness enhancement method according to any one of claims 1 to 10 or a training method for enhancing a parametric perceptual model according to claim 11.