CN115334228B - Video processing method and related device - Google Patents

Abstract

The application discloses a video processing method which is applied to a terminal. The method comprises the following steps: when the terminal is in a night scene shooting mode, detecting whether a lamplight area exists in an image shot by the terminal; when a lamplight area exists in an image shot by a terminal, determining a first image in an image sequence acquired by the terminal, wherein the adjacent area of the lamplight area in the first image is overexposed; determining a second image and a third image in the image sequence according to the first image, wherein the overexposure degree of the adjacent areas of the light areas in the second image and the third image is smaller than that of the adjacent areas of the light areas in the first image; generating a target image according to the first image, the second image and the third image; based on the target image, a video is generated. Based on the scheme, the tremble image in the video can be replaced by the image which is normally displayed, so that the problem that the light area in the video picture is abnormally widened due to lens shake is solved.

Description

A video processing method and related device

Technical Field

The present application relates to the field of video processing technology, and in particular to a video processing method and related devices.

Background Art

With the development of society, people are increasingly using various terminals to shoot videos, including in the consumer field and video surveillance field. The lens shake problem when shooting videos with portable terminal devices such as smartphones has always been one of the biggest difficulties in generating high-quality videos. Since portable terminals are relatively light, users usually use portable terminals to shoot videos while moving, resulting in unstable video images. During the user's movement, the lens of the portable terminal will shake, causing the light area in the video image to be sensed by the photosensitive element in a larger area, resulting in the abnormal expansion of the light area in the video image.

Currently, there is an urgent need for a video processing solution to solve the problem of abnormal expansion of the light area in the video image caused by lens shaking.

Summary of the invention

The present application provides a video processing method, which can replace the shaky image in the video with a normally displayed image, thereby solving the problem of abnormal expansion of the light area in the video screen caused by lens shaking.

The first aspect of the present application provides a video processing method, which can be applied to a terminal. The method comprises: when the terminal is in a night scene shooting mode, detecting whether there is a light area in an image shot by the terminal.

When there is a light area in the image captured by the terminal, the terminal determines a first image in the acquired image sequence, and an area adjacent to the light area in the first image is overexposed. The overexposure of the area adjacent to the light area in the first image means that the brightness value of the pixels in the adjacent area is too high. The reason why the area adjacent to the light area in the first image is overexposed is that the terminal has a jitter displacement when acquiring the first image, so that the photosensitive element corresponding to the adjacent area in the terminal also senses the light in the light area, which ultimately causes the area adjacent to the light area in the first image to be overexposed.

After determining the first image, the terminal determines the second image and the third image in the image sequence according to the first image, the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image, and the overexposure degree of the adjacent area of the light area in the second image and the third image is less than the overexposure degree of the adjacent area of the light area in the first image. Among them, the light area in the second image and the third image has a corresponding relationship with the light area in the first image, that is, the light area in the second image and the third image represents the same picture content as the light area in the first image. For example, the light area in the second image and the third image represents the light-emitting area of a street lamp; the light area in the first image also represents the light-emitting area of the same street lamp.

The terminal generates a target image based on the first image, the second image, and the third image, the target image is used to replace the first image, and the overexposure degree of the neighboring area of the light area of the target image is smaller than the overexposure degree of the neighboring area of the light area of the first image. The second image and the third image are used as reference images to generate a target image to replace the first image, so that the overexposure degree of the neighboring area of the light area of the generated target image is smaller than the overexposure degree of the neighboring area of the light area of the first image.

The terminal generates a video based on the target image. Specifically, the terminal may replace the first image in the acquired image sequence with the target image, thereby generating a video.

In this embodiment, by locating the overexposed trembling image in the vicinity of the light area in the night scene shooting mode, the images before and after the trembling image are used as reference images, and a target image for replacing the trembling image is generated based on the trembling image and the reference image. Through this solution, the trembling image in the video can be replaced with a normally displayed image, thereby solving the problem of abnormal expansion of the bright part of the light area in the video screen caused by lens shaking.

In a possible implementation, the terminal may detect whether there is a light area in the captured image by means of image recognition or a pre-trained neural network.

Specifically, the terminal can determine whether there is a light area in the image by detecting the brightness value of the pixels in the captured image. Specifically, the brightness values of the pixels in the light area are all greater than or equal to a specific threshold, and the difference between the brightness value of the pixels in the light area and the brightness value of the pixels in the adjacent area of the light area is greater than or equal to a preset difference.

The value of the above-mentioned specific threshold can be 180-230, for example, the specific threshold is specifically 224; the value of the preset difference can be 25-50, for example, the preset difference is 32. For example, when the terminal detects that the brightness values of pixels in a certain area in the image are greater than or equal to 224, and the difference between the brightness values of the pixels in the area and the brightness values of the pixels in the adjacent area is greater than or equal to 32, the terminal can determine that the area is a light area. The adjacent area of the light area can be composed of pixels adjacent to the pixels in the light area, that is, the pixels in the adjacent area of the light area are adjacent to the pixels in the light area.

In addition, after the terminal determines that there is a light area in the image, the terminal can also determine the position of the light area in the image according to the coordinates of the pixels located in the light area. The position of the light area in the image can be determined by the coordinates of the pixels at the boundary of the light area, that is, the area surrounded by the pixels at the boundary of the light area is the light area. The terminal can determine the position of the light area by recording the coordinates of the pixels at the boundary of the light area or the coordinates of all pixels in the light area.

Specifically, the degree of overexposure of the areas adjacent to the light area in the second image and the third image is less than the degree of overexposure of the areas adjacent to the light area in the first image, which may mean that: the area of the overexposed area in the areas adjacent to the light area in the second image and the third image is smaller than the area of the overexposed area in the areas adjacent to the light area in the first image; or, the brightness value of the overexposed pixels in the areas adjacent to the light area in the second image and the third image is smaller than the brightness value of the overexposed pixels in the areas adjacent to the light area in the first image.

Optionally, the area of the overexposed region in the second image and the adjacent region of the light region in the third image may be 0, that is, the adjacent region of the light region in the second image and the third image is not overexposed. In general, compared with the first image, the second image and the third image are captured by the terminal with less jitter displacement or no jitter displacement, so the overexposure degree of the adjacent region of the light region in the second image and the third image is less than the overexposure degree of the adjacent region of the light region in the first image.

In a possible implementation, the method further includes: when the illumination of the ambient light is less than a first preset threshold, the terminal enters a night scene shooting mode. Exemplarily, the value of the first preset threshold may be 30-60, for example, the first preset threshold is 50. During the shooting process, the terminal may detect the illumination of the ambient light, and when the illumination of the ambient light is less than 50, the terminal enters the night scene shooting mode. Or, when the terminal obtains an instruction for triggering entry into the night scene shooting mode, the terminal enters the night scene shooting mode. For example, in the process of the user operating the terminal to shoot, the user may send an instruction to the terminal to enter the night scene shooting mode by touching the terminal screen; in this way, when the terminal obtains an instruction for triggering entry into the night scene shooting mode, the terminal enters the night scene shooting mode.

In addition, the terminal can also determine whether the current shooting scene is a night scene according to the picture content of the preview image during shooting and/or the ambient brightness value of each area of the preview image, thereby determining whether to enter the night scene shooting mode. For example, when the picture content of the preview image includes the night sky or night scene light sources, the terminal can determine that the current shooting scene is a night scene, thereby entering the night scene shooting mode; or, when the ambient brightness value of each area of the preview image meets the brightness distribution characteristics of the image in the night scene environment, the terminal can determine that the current shooting scene is a night scene, thereby entering the night scene shooting mode.

In a possible implementation, the jitter displacement when the terminal collects the first image is greater than a second preset threshold. The second preset threshold may be 300-400, for example, the second preset threshold is specifically 360. The jitter displacement when the terminal collects the first image may be referred to as the jitter value of the first image. Specifically, the terminal may obtain sensor data that records the movement of the terminal when collecting the first image, and calculate the jitter value of the first image based on the sensor data. The terminal may also calculate the contrast value of the image, and determine the jitter value of the image based on the contrast value of the image, that is, the smaller the contrast value of the image, the larger the jitter value of the image. In addition, the terminal may also determine the jitter value of the image based on a pre-trained neural network.

In a possible implementation, the terminal may also determine whether there is a local light area in the multiple images. Only when a certain image has a local light area and the jitter value of the image is greater than or equal to a second preset threshold, the terminal determines the image as a trembling image.

Wherein, there is a local light area in the first image, the ratio of the number of target pixels in the first image to the total number of pixels in the first image is within a preset range, and the target pixel is a pixel in the light area in the first image. That is, the ratio of the area of the light area in the first image to the area of all areas in the first image is within a preset range. The value of the preset range can be, for example, 2%-20%, that is, the area of the light area in the first image accounts for 2%-20% of the area in the first image. Exemplarily, the terminal can determine the number of target pixels in each of the multiple images, the target pixel refers to the pixel in the light area, and the definition of the light area can refer to the above description. Then, the terminal obtains the ratio of the number of target pixels in each image to the total number of pixels in the image, and determines whether the ratio is within the preset range. If the ratio of the number of target pixels in the image to the total number of pixels in the image is within the preset range, the terminal determines that there is a local light area in the image; if the ratio of the number of target pixels in the image to the total number of pixels in the image is not within the preset range, the terminal determines that there is no local light area in the image.

In a possible implementation, the terminal determines the second image and the third image from the multiple images based on the first image, including the following process: the terminal determines the stability value of each of the multiple images respectively, and the stability value is the reciprocal of the sum of the jitter value of the image and the acquisition time difference between the image and the first image. Among them, the stability value of the image is negatively correlated with the jitter value of the image and the acquisition time difference between the image and the first image. In other words, the larger the jitter value of the image, the smaller the stability value of the image; the larger the acquisition time difference between the image and the first image, the smaller the stability value of the image. In short, the terminal needs to select images with small jitter values and close to the first image as the second image and the third image as much as possible.

Then, the terminal determines that the image with the largest stability value among the multiple images is the second image, and determines the third image among the multiple images based on the acquisition time of the second image, so that the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image.

Specifically, the terminal can determine one or more images from the multiple images according to the acquisition time of the second image, and the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the one or more images; then, the terminal determines the image with the largest stability value from the one or more images as the third image. Based on the stability value of the second image determined in the above manner is greater than that of the third image, the second image can be considered as the primary reference image for generating the target image, and the third image can be considered as the secondary reference image for generating the target image.

In a possible implementation, the terminal determines the motion vectors corresponding to the multiple image blocks respectively according to the first image and the third image, including the following process: First, the terminal obtains multiple candidate motion vectors corresponding to the first image block. The multiple image blocks in the second image include the first image block, and the first image block can be any one of the multiple image blocks. Then, the terminal determines the second image block and the third image block corresponding to each candidate motion vector in the multiple candidate motion vectors according to the position of the first image block, the first image includes the second image block, and the third image includes the third image block.

Secondly, the terminal determines a target error value corresponding to each candidate motion vector according to the first image block, the second image block and the third image block, wherein the target error value is obtained based on an error value between the first image block and the second image block and an error value between the first image block and the third image block. The error value between two image blocks represents a difference between the two image blocks, and the larger the error value, the larger the difference between the two image blocks.

Finally, the terminal determines, according to the target error value corresponding to each candidate motion vector, a motion vector with the smallest target error value among the multiple candidate motion vectors as the motion vector corresponding to the first image block.

In a possible implementation manner, the multiple candidate motion vectors include: one or more preset motion vectors, one or more randomly generated motion vectors and/or motion vectors corresponding to an image block adjacent to the first image block.

Since the second image and the third image are two images captured at similar times, the displacement of the object in the second image relative to the third image is small, so it can be considered that the motion vector corresponding to the image block in the second image is within the preset range. In this way, the terminal can obtain one or more preset motion vectors and use these preset motion vectors as candidate motion vectors. In addition, in order to ensure that the motion vector actually corresponding to the first image block can be obtained as much as possible, one or more motion vectors can be randomly generated or the motion vector corresponding to the image block adjacent to the first image block can be selected as the candidate motion vector, so as to avoid the value range of the candidate motion vector being too limited.

In one possible implementation, after the terminal generates a target image based on multiple image blocks of the second image, if there is an area in the target image where an image cannot be generated based on the multiple image blocks, the terminal divides the third image to obtain multiple image blocks of the third image.

Then, the terminal determines the motion vectors corresponding to the multiple image blocks of the third image respectively according to the first image and the second image. The process of the terminal determining the motion vectors corresponding to the multiple image blocks of the third image respectively is similar to the process of the terminal determining the motion vectors corresponding to the multiple image blocks of the second image. For details, please refer to the description of the above embodiment, which will not be repeated here.

Finally, the terminal updates the target image according to the motion vectors corresponding to the multiple image blocks of the third image and the multiple image blocks of the third image, and obtains a new target image, and the new target image is used to replace the first image. The terminal updates the target image means that the terminal moves the image blocks in the third image to the hole area in the target image according to the motion vectors corresponding to the multiple image blocks of the third image, thereby filling the hole area. For areas in the target image where images already exist, these areas where images already exist are no longer updated.

In this solution, by obtaining the motion vector of the third image and updating the target image based on the motion vector of the third image, the hole area in the target image can be effectively eliminated and the image quality of the target image can be improved.

In a possible implementation, the terminal obtains sensor data corresponding to the first image according to the exposure time of the first image, and the sensor data is used to record the movement of the terminal during the exposure time of the first image. The sensor data may be, for example, gyroscope data, and the gyroscope data may record the angular velocity of the terminal during the exposure time of the first image. After obtaining the sensor data, the terminal determines the jitter value of the first image according to the sensor data, that is, determines the jitter displacement of the terminal during the exposure period of the first image based on the sensor data.

A second aspect of the present application provides a video processing device, including:

A detection unit, used to detect whether there is a light area in the image captured by the terminal when the terminal is in a night scene shooting mode;

A first determining unit is configured to, when there is a light area in the image captured by the terminal, determine a first image in the image sequence collected by the terminal, in which an area adjacent to the light area in the first image is overexposed;

a second determining unit, configured to determine a second image and a third image in the image sequence according to the first image, wherein a capture time of the first image is between a capture time of the second image and a capture time of the third image, and an overexposure degree of an area adjacent to the light area in the second image and the third image is smaller than an overexposure degree of an area adjacent to the light area in the first image;

The image generating unit is further used to generate a target image according to the first image, the second image and the third image, wherein the target image is used to replace the first image, and the overexposure degree of the neighboring area of the light area of the target image is smaller than the overexposure degree of the neighboring area of the light area of the first image;

The video generation unit is further used to generate a video based on the target image.

In a possible implementation, the device further includes: a control unit;

The control unit is used to: when the illumination of the ambient light is less than a first preset threshold, control the terminal to enter a night scene shooting mode; or, when an instruction for triggering entering a night scene shooting mode is obtained, control the terminal to enter a night scene shooting mode.

In a possible implementation manner, a jitter displacement when the terminal captures the first image is greater than a second preset threshold.

In a possible implementation manner, a ratio of the number of target pixels in the first image to the total number of pixels in the first image is within a preset range, and the target pixels are pixels in a light area in the first image.

In a possible implementation, the second determination unit is used to: separately determine a stability value of each of the multiple images, where the stability value is the inverse of the sum of a jitter value of the image and an acquisition time difference between the image and the first image; determine the image with the largest stability value among the multiple images as the second image; and determine the third image among the multiple images based on the acquisition time of the second image.

In a possible implementation, the second determination unit is used to: determine one or more images from the multiple images based on the acquisition time of the second image, the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the one or more images; and determine the image with the largest stability value from the one or more images as the third image.

In a possible implementation, the image generation unit is used to: divide the second image into multiple image blocks; determine the motion vectors corresponding to the multiple image blocks respectively according to the first image and the third image; and generate the target image according to the motion vectors corresponding to the multiple image blocks respectively and the multiple image blocks.

In a possible implementation, the image generation unit is used to: obtain multiple candidate motion vectors corresponding to a first image block, the multiple image blocks including the first image block; determine, according to the position of the first image block, a second image block and a third image block corresponding to each of the multiple candidate motion vectors, the first image including the second image block, and the third image including the third image block; determine, according to the first image block, the second image block, and the third image block, a target error value corresponding to each candidate motion vector, the target error value being obtained based on an error value between the first image block and the second image block and an error value between the first image block and the third image block; and, according to the target error value corresponding to each candidate motion vector, determine, among the multiple candidate motion vectors, a motion vector with a minimum target error value as the motion vector corresponding to the first image block.

In a possible implementation manner, the multiple candidate motion vectors include: one or more preset motion vectors, one or more randomly generated motion vectors and/or motion vectors corresponding to an image block adjacent to the first image block.

In a possible implementation, the image generation unit is used to: if there is an area in the target image where an image cannot be generated based on the multiple image blocks, divide the third image to obtain multiple image blocks of the third image; determine motion vectors corresponding to the multiple image blocks of the third image according to the first image and the second image; update the target image according to the motion vectors corresponding to the multiple image blocks of the third image and the multiple image blocks of the third image to obtain a new target image, and the new target image is used to replace the first image.

The third aspect of the present application provides a video processing device, which includes: a processor, a non-volatile memory and a volatile memory; wherein computer-readable instructions are stored in the non-volatile memory or the volatile memory; the processor reads the computer-readable instructions to enable the video processing device to implement a method such as any one of the implementation methods in the first aspect.

According to a fourth aspect of the present application, there is provided a terminal device, comprising a processor, a memory, a display screen, a camera and a bus, wherein: the processor, the display screen, the camera and the memory are connected via the bus; the memory is used to store computer programs and images captured by the camera; the processor is used to control the display screen and the memory, obtain the images stored in the memory and execute the programs stored in the memory to perform the following steps: when the terminal is in a night scene shooting mode, detect whether there is a light area in the image captured by the terminal; when there is a light area in the image captured by the terminal, determine a first image in the image sequence captured by the terminal, and the first image in the image sequence captured by the terminal. An area adjacent to a light area in an image is overexposed; a second image and a third image are determined in the image sequence based on the first image, the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image, and the degree of overexposure of areas adjacent to the light area in the second image and the third image is less than the degree of overexposure of areas adjacent to the light area in the first image; a target image is generated based on the first image, the second image and the third image, the target image is used to replace the first image, and the degree of overexposure of areas adjacent to the light area in the target image is less than the degree of overexposure of areas adjacent to the light area in the first image; and a video is generated based on the target image.

In one possible implementation, the processor is also used to: when the illumination of the ambient light is less than a first preset threshold, the terminal enters a night scene shooting mode; or, when the terminal obtains an instruction for triggering entry into the night scene shooting mode, the terminal enters the night scene shooting mode.

In a possible implementation manner, a jitter displacement when the terminal captures the first image is greater than a second preset threshold.

In a possible implementation manner, a ratio of the number of target pixels in the first image to the total number of pixels in the first image is within a preset range, and the target pixels are pixels in a light area in the first image.

In a possible implementation, the processor is also used to: separately determine a stability value of each of the multiple images, where the stability value is the inverse of the sum of the jitter value of the image and the acquisition time difference between the image and the first image; determine the image with the largest stability value among the multiple images as the second image; and determine the third image among the multiple images based on the acquisition time of the second image.

In a possible implementation, the processor is further used to: determine one or more images from the multiple images based on the acquisition time of the second image, the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the one or more images; and determine the image with the largest stability value from the one or more images as the third image.

In a possible implementation, the processor is further used to: divide the second image into multiple image blocks; determine the motion vectors corresponding to the multiple image blocks respectively according to the first image and the third image; and generate the target image according to the motion vectors corresponding to the multiple image blocks respectively and the multiple image blocks.

In a possible implementation, the processor is further used to: obtain multiple candidate motion vectors corresponding to a first image block, the multiple image blocks including the first image block; determine, based on the position of the first image block, a second image block and a third image block corresponding to each of the multiple candidate motion vectors, the first image including the second image block, and the third image including the third image block; determine, based on the first image block, the second image block, and the third image block, a target error value corresponding to each candidate motion vector, the target error value being obtained based on an error value between the first image block and the second image block and an error value between the first image block and the third image block; and, based on the target error value corresponding to each candidate motion vector, determine, among the multiple candidate motion vectors, a motion vector with a minimum target error value as the motion vector corresponding to the first image block.

In a possible implementation manner, the multiple candidate motion vectors include: one or more preset motion vectors, one or more randomly generated motion vectors and/or motion vectors corresponding to an image block adjacent to the first image block.

In a possible implementation, the processor is further used to: if there is an area in the target image where an image cannot be generated based on the multiple image blocks, divide the third image to obtain multiple image blocks of the third image; determine the motion vectors corresponding to the multiple image blocks of the third image according to the first image and the second image; and update the target image according to the motion vectors corresponding to the multiple image blocks of the third image and the multiple image blocks of the third image to obtain a new target image, and the new target image is used to replace the first image.

A fifth aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes a method as in any one of the implementation modes of the first aspect.

A sixth aspect of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method as in any one of the implementation modes of the first aspect.

In a seventh aspect, the present application provides a chip, comprising one or more processors, wherein some or all of the processors are used to read and execute a computer program stored in a memory to execute the method in any possible implementation of any of the above aspects.

Optionally, the chip includes a memory, and the memory is connected to the processor through a circuit or wire. Optionally, the chip also includes a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive data and/or information to be processed, and the processor obtains the data and/or information from the communication interface, processes the data and/or information, and outputs the processing results through the communication interface. The communication interface can be an input and output interface. The method provided in the present application can be implemented by one chip, or by multiple chips in collaboration.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1a is a schematic diagram showing a comparison between a tremor image and an image used to replace the tremor image provided by an embodiment of the present application;

FIG1b is a schematic diagram showing a comparison between another tremor image and an image used to replace the tremor image provided in an embodiment of the present application;

FIG2a is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

FIG2b is a software structure block diagram of the terminal 100 provided in an embodiment of the present application;

FIG3 is a flow chart of a video processing method provided in an embodiment of the present application;

FIG4 is a schematic diagram of a MEMC method provided in an embodiment of the present application;

FIG5 is another schematic diagram of a MEMC method provided in an embodiment of the present application;

FIG6 is a schematic diagram of determining an image block based on a candidate motion vector provided by an embodiment of the present application;

FIG7 is a schematic diagram of generating a target image provided by an embodiment of the present application;

FIG8 is a schematic diagram of another method for generating a target image provided by an embodiment of the present application;

FIG9 is a schematic diagram of an application architecture of a video processing method provided in an embodiment of the present application;

FIG10 is a schematic diagram of a process of replacing a tremor image provided by an embodiment of the present application;

FIG11 is another schematic diagram of a process of replacing a tremor image provided by an embodiment of the present application;

FIG12 is a schematic diagram of a flow chart of another video processing method provided in an embodiment of the present application;

FIG13 is a schematic diagram of the structure of a terminal 1300 provided in an embodiment of the present application;

FIG. 14 is a schematic diagram of the structure of a computer program product 1400 provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time/logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

For a long time, the lens shake problem when shooting videos with portable terminal devices such as smartphones has been one of the biggest difficulties in generating high-quality videos. Since portable terminals are relatively light, users usually move while shooting videos with portable terminals, resulting in unstable video images taken by portable terminals. During the user's movement, the lens of the portable terminal will shake, causing the light area in the video image to be sensed by the photosensitive element in a larger area, resulting in the abnormal expansion of the light area in the video image.

During the user's movement, the lens of the portable terminal usually shakes to varying degrees, so the light area of some video frames in the video captured by the portable terminal expands abnormally, while the light area of another part of the video frames in the video is displayed normally. During the user's video viewing, video frames with abnormally expanded light areas and video frames with normal light areas appear alternately, which can easily give the user a subjective sense of tremor and affect the user's experience. Among them, the video frame with abnormally expanded light areas in the picture can be called a tremor image.

At present, the electronic image stabilization (EIS) technology configured in portable terminals can rotate, translate and correct the video image to keep the objects in the video image in a stable position. However, the existing EIS technology cannot handle the problem of abnormal expansion of the light area in the video image. Based on the EIS technology, the display position of the light area in the video is stable in the stable frame and the shaking frame, but the phenomenon of abnormal expansion of the light area still occurs in the shaking frame.

In view of this, an embodiment of the present application provides a video processing method, which locates a trembling image in a video by determining the jitter value of an image in the video. Then, the images before and after the trembling image are used as reference images, and the relative displacement between image blocks in the reference image is obtained based on the trembling image and the reference image. Finally, a target image for replacing the trembling image is generated based on the image blocks in the reference image and the relative displacement corresponding to the image blocks. Through this solution, the trembling image in the video can be replaced with a normally displayed image, thereby solving the problem of abnormal expansion of the light area in the video screen caused by lens shaking.

Please refer to Figures 1a and 1b, Figure 1a is a schematic diagram of the comparison between the tremor image provided in an embodiment of the present application and an image used to replace the tremor image; Figure 1b is a schematic diagram of the comparison between another tremor image provided in an embodiment of the present application and an image used to replace the tremor image.

As shown in FIG. 1a, the trembling image 1 in FIG. 1a is an image captured in an outdoor scene. As can be seen from FIG. 1a, the light area in the trembling image 1, that is, the area where the windows of the building are located, has an obvious outward expansion phenomenon. After the trembling image 1 is processed by the method provided in the embodiment of the present application, a stable image 1 is obtained to replace the trembling image 1, and the light area in the stable image 1 is displayed normally.

As shown in FIG1b, the trembling image 2 in FIG1b is an image captured in an indoor scene. As can be seen from FIG1b, the light area in the trembling image 2, that is, the area where the lamps on the indoor ceiling are located, has an obvious outward expansion phenomenon. After processing the trembling image 2, a stable image 2 is obtained to replace the trembling image 2, and the light area in the stable image 2 is displayed normally.

The video processing method provided in the embodiment of the present application can be applied to a terminal with a video acquisition function. The terminal is also called user equipment (UE), mobile station (MS), mobile terminal (MT), etc., which is a device equipped with a terminal capable of shooting video, and can process the captured video to output a stable displayed video. For example, a handheld device with a shooting function, a surveillance camera, etc.

At present, some examples of terminals are: mobile phones, tablet computers, laptops, PDAs, surveillance cameras, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc.

The image acquisition device in the terminal is used to convert the optical signal into an electrical signal to generate an image signal. The image acquisition device may be, for example, an image sensor, which may be, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

For ease of understanding, the structure of the terminal 100 provided in the embodiment of the present application is described below by way of example. Referring to Fig. 2a, Fig. 2a is a schematic diagram of the structure of the terminal device provided in the embodiment of the present application.

As shown in Figure 2a, the terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal 100.

The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal 100.

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal 100, and can also be used to transmit data between the terminal 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The charging management module 140 is used to receive charging input from a charger.

The power management module 141 is used to connect the battery 142 , the charging management module 140 and the processor 110 .

The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 150 can provide solutions for wireless communications including 2G/3G/4G/5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

In some embodiments, the antenna 1 of the terminal 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and/or a satellite based augmentation system (SBAS).

The terminal 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

Specifically, the display screen 194 can display the target interface in this embodiment.

The terminal 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture a still image or a video.

Video codecs are used to compress or decompress digital videos. Terminal 100 may support one or more video codecs. Thus, terminal 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal 100 by running instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The terminal 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal 100 can listen to music or listen to a hands-free call through the speaker 170A. The receiver 170B, also called a "handset", is used to convert an audio electrical signal into a sound signal. The microphone 170C, also called a "microphone" or "microphone", is used to convert a sound signal into an electrical signal. The headphone jack 170D is used to connect a wired headphone.

The pressure sensor 180A is used to sense pressure signals and convert the pressure signals into electrical signals.

The gyro sensor 180B may be used to determine a motion posture of the terminal 100 .

The air pressure sensor 180C is used to measure air pressure.

The magnetic sensor 180D includes a Hall sensor.

The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal 100 in various directions (generally three axes).

The distance sensor 180F is used to measure the distance.

The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode.

The ambient light sensor 180L is used to sense the brightness of ambient light.

The fingerprint sensor 180H is used to collect fingerprints.

The temperature sensor 180J is used to detect the temperature.

The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal 100, which is different from the position of the display screen 194.

Bone conduction sensor 180M can obtain vibration signals.

The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal 100 may receive key input and generate key signal input related to user settings and function control of the terminal 100.

Motor 191 can generate vibration prompts.

Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

The SIM card interface 195 is used to connect a SIM card.

The software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the terminal 100.

FIG. 2 b is a software structure block diagram of the terminal 100 provided in an embodiment of the present application.

The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

The application layer can include a series of application packages.

As shown in FIG. 2 b , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

As shown in FIG. 2 b , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

The phone manager is used to provide communication functions of the terminal 100, such as management of call status (including connection, disconnection, etc.).

The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

A 2D graphics engine is a drawing engine for 2D drawings.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

The following is an illustrative description of the software and hardware workflow of the terminal 100 in conjunction with a video playback scenario.

When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, timestamp of the touch operation and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer and identifies the controls corresponding to the input events. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is a control of a video playback-related application icon, the video playback application calls the interface of the application framework layer to start the video playback application, and then plays the video in the video playback interface of the video playback application by calling the kernel layer, for example, a free viewpoint video can be played.

The above introduces the application scenario of the video processing method provided in the embodiment of the present application. The following will introduce the execution process of the video processing method in detail.

Please refer to Figure 3, which is a schematic flow chart of a video processing method provided by an embodiment of the present application. As shown in Figure 3, the video processing method includes the following steps 301-304.

Step 301, acquiring multiple images.

In this embodiment, the multiple images may be multiple continuous images in a video. The terminal may obtain the multiple images in multiple ways.

In one possible manner, the terminal may shoot a video at a fixed frame rate to obtain multiple images shot within a period of time. For example, when the terminal shoots a video at a frame rate of 30 Hz, the multiple images may be 30 images shot by the terminal within 1 second.

In another possible manner, the terminal may receive multiple images sent by other terminals, where the multiple images are captured by other terminals. In short, the other terminals capture video at a fixed frame rate, and after capturing multiple images, the multiple images are sent to the terminal in this embodiment, which processes the multiple images.

Step 302: determine a first image among the multiple images, wherein a jitter value of the first image is greater than or equal to a second preset threshold, and the jitter value is a jitter displacement when the terminal captures the first image.

In this embodiment, the terminal determines the trembling image in the multiple images by determining the jitter value of each image in the multiple images. When the jitter value of any one of the multiple images is less than the second preset threshold, it is determined that the image is not a trembling image; when the jitter value of one of the multiple images is greater than or equal to the second preset threshold, it is determined that the image is a trembling image. The jitter value of the image refers to the jitter displacement of the terminal when acquiring the image. The second preset threshold can be 300-400, for example, the second preset threshold is specifically 360.

In a possible embodiment, the terminal may obtain sensor data recording a movement condition of the terminal when capturing the first image, and calculate a jitter value of the first image based on the sensor data.

Exemplarily, the terminal obtains sensor data corresponding to the first image according to the exposure time of the first image, and the sensor data is used to record the movement of the terminal during the exposure time of the first image. For example, the sensor data may be gyroscope data, that is, data collected by a gyroscope. Among them, the gyroscope is an angular motion detection device, and the gyroscope data collected by the gyroscope records the angular velocity of the terminal during the exposure time of the first image. In addition, the sensor data may also be data collected by other sensors, and the other sensors may record the angular velocity or angular acceleration of the terminal during the exposure time of the first image. Then, the terminal determines the jitter value of the first image according to the sensor data, that is, determines the jitter displacement of the terminal during the exposure of the first image.

For example, when the sensor data is gyroscope data, the terminal can obtain gyroscope data at multiple moments during the exposure of the first image, and each gyroscope data records the three-dimensional rotation angular velocity of the terminal at a certain moment. Therefore, the terminal can calculate the angular displacement of the terminal between two adjacent acquisition moments based on the multiple gyroscope data during the exposure of the first image and the time interval between two adjacent gyroscope data. Finally, the terminal determines the jitter displacement of the terminal during the exposure of the first image based on the calculated multiple angular displacements.

Based on the above method, the terminal can obtain the jitter value of each image in the multiple images, thereby determining the first image whose jitter value is greater than or equal to the second preset threshold.

In this embodiment, in addition to calculating the jitter value of the image through sensor data, the terminal may also obtain the jitter value of the image through other methods.

In a possible implementation manner, the terminal may calculate a contrast value of the image, and determine a jitter value of the image according to the contrast value of the image.

Specifically, for the multiple images, the terminal can calculate the contrast value of each image in the multiple images, and then determine the jitter value of each image based on the contrast value of each image. Among them, the terminal can calculate the contrast value of the image based on the brightness value of the pixels in the region of interest of the image. Specifically, the terminal can first determine the region of interest in the image, and the region of interest can be, for example, a region of a specific range located in the center of the image. For example, the region of interest is the region where 12*12 pixels in the center of the image are located. Then, the terminal calculates the brightness values of all pixels in the region of interest, and determines the pixel with the highest brightness and the pixel with the lowest brightness in the region of interest. Finally, the terminal calculates the ratio between the brightness value of the pixel with the highest brightness and the brightness value of the pixel with the lowest brightness, that is, the terminal calculates the ratio between the highest brightness value and the lowest brightness value, thereby obtaining the contrast value of the image.

After calculating the contrast value of the image, the terminal further determines the jitter value of the image according to the contrast value of the image. Among them, the jitter value of the image is negatively correlated with the contrast value of the image. That is, the larger the contrast value of the image, the smaller the jitter value of the image; the smaller the contrast value of the image, the larger the jitter value of the image. Exemplarily, the product of the jitter value of the image and the contrast value of the image can be defined as a specific positive number, so the terminal can determine the jitter value of the image by dividing the positive number by the contrast value of the image. For example, assuming that the contrast value of the image is C, the terminal can calculate the jitter value of the image as 1/C.

In another possible implementation manner, the terminal may determine the jitter value of the image based on a pre-trained neural network.

Exemplarily, the terminal can obtain a neural network that has been pre-trained based on a large number of images marked with jitter values, and the neural network is used to predict the jitter value of the image. Then, the terminal inputs the multiple images into the neural network to obtain the jitter value output by the neural network, thereby determining the jitter value of each of the multiple images. Among them, the neural network is trained based on pre-prepared training data. The training data includes a large number of images taken when the terminal is jittering, and the light areas in these images all have abnormal outward expansion. In addition, these images are marked with the jitter values corresponding to the images. After obtaining the training data, the neural network can be trained according to the existing neural network training method, and finally a trained neural network is obtained.

Based on the above two implementation methods, the terminal can effectively determine whether the image is a tremor image when the terminal cannot obtain the corresponding sensor data when collecting the image.

In a possible embodiment, the terminal may further determine whether there is a local light area in the multiple images. Only when a certain image has a local light area and the jitter value of the image is greater than or equal to a second preset threshold, the terminal determines the image as a trembling image.

In this embodiment, the brightness values of the pixels in the light area are all greater than a specific threshold, and the difference between the brightness values of the pixels in the light area and the brightness values of the pixels in the adjacent area of the light area is greater than a preset difference. The value of the above-mentioned specific threshold can be 180-230, for example, the specific threshold is specifically 224; the value of the preset difference can be 25-50, for example, the preset difference is 32. For example, when the terminal detects that the brightness values of the pixels in a certain area of the image are all greater than 224, and the difference between the brightness values of the pixels in the area and the brightness values of the pixels in the adjacent area is greater than 32, the terminal can determine that the area is a light area. The adjacent area of the light area can be composed of pixels adjacent to the pixels in the light area, that is, the pixels in the adjacent area of the light area are adjacent to the pixels in the light area.

Exemplarily, the terminal may determine the number of target pixels in each of the multiple images, where the target pixels refer to pixels in the light area. Then, the terminal obtains the ratio of the number of target pixels in each image to the total number of pixels in the image, and determines whether the ratio is within a preset range. If the ratio of the number of target pixels in the image to the total number of pixels in the image is within the preset range, the terminal determines that there is a local light area in the image; if the ratio of the number of target pixels in the image to the total number of pixels in the image is not within the preset range, the terminal determines that there is no local light area in the image.

That is, the jitter value of the first image determined as a trembling image is greater than or equal to a second preset threshold, and the ratio of the number of target pixels to the total number of pixels in the first image is within a preset range, and the target pixels are pixels in the light area.

Exemplarily, taking the YUV format as an example, the Y value of a pixel in the image is the brightness value of the pixel. In the image, the Y value of a pixel ranges from 0 to 255, and the larger the Y value, the larger the brightness value of the pixel. Assume that the above-mentioned specific threshold is 224, the preset difference is 32, and the preset range is 2%-20%. Then, the terminal can first determine the number of target pixels in the image, and then the terminal determines the ratio of the target pixel to the total number of pixels in the image. Among them, the Y value of the target pixel is greater than or equal to 240, and the difference between the brightness value of the target pixel and the brightness value of the pixel in the adjacent area of the light area is greater than or equal to 32. If the ratio of the target pixel to the total number of pixels in the image is within 2%-20%, it is determined that the image has a local light area. It can be understood that the above-mentioned threshold, difference and preset range can be adjusted according to actual needs, and this embodiment does not limit the specific values of the threshold, difference and preset range.

Step 303: Determine a second image and a third image from the multiple images according to the first image.

In this embodiment, the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image. That is, among the multiple images acquired by the terminal, the first image is not the first image among the multiple images, and the first image is not the last image among the multiple images. Among them, the multiple images are arranged in the order of acquisition time, the first image among the multiple images refers to the earliest acquired image among the multiple images, and the last image among the multiple images refers to the latest acquired image among the multiple images.

In addition, the jitter value of at least one of the second image and the third image is smaller than the jitter value of the first image. Since the second image and the third image are used as reference images to generate a target image that replaces the first image, the image quality of the second image and the third image needs to be guaranteed so as to generate a target image in which the light area is normally displayed.

In a possible implementation, the terminal may be centered on the first image and determine the second image and the third image from N images located before and after the first image. Exemplarily, the terminal may obtain k images located before the first image at the acquisition time and k images located after the first image at the acquisition time from the aforementioned multiple images, and determine the second image and the third image from the acquired 2k images. Wherein, k is an integer greater than or equal to 1, for example, k may be 1, 2 or 3. Since the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image, the terminal may determine the second image from the k images located before the first image at the acquisition time, and then determine the third image from the k images located after the first image at the acquisition time. Alternatively, the terminal may determine the third image from the k images located before the first image at the acquisition time, and then determine the second image from the k images located after the first image at the acquisition time.

For example, assuming that the first image is image t, the terminal can obtain images t-k to image t-1 and images t+1 to image t+k, and determine the second image and the third image in images t-k to image t-1 and images t+1 to image t+k. Then, the terminal can determine the second image in images t-k to image t-1, and determine the third image in images t+1 to image t+k. Alternatively, the terminal can determine the third image in images t-k to image t-1, and determine the second image in images t+1 to image t+k.

Optionally, in order to generate a target image with higher quality as much as possible, the terminal may select images with greater stability values for generating a high-quality target image from multiple images as the second image and the third image.

Exemplarily, the terminal determines the stability value of each of the multiple images respectively, and the stability value is the inverse of the sum of the jitter value of the image and the acquisition time difference between the image and the first image. Among them, the stability value of the image is negatively correlated with the jitter value of the image and the acquisition time difference between the image and the first image. In other words, the larger the jitter value of the image, the smaller the stability value of the image; the larger the acquisition time difference between the image and the first image, the smaller the stability value of the image. In short, the terminal needs to select images with small jitter values and close to the first image as the second image and the third image as much as possible.

Then, the terminal determines that the image with the largest stability value among the multiple images is the second image, and determines the third image among the multiple images according to the acquisition time of the second image, so that the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image. Specifically, the terminal can determine one or more images among the multiple images according to the acquisition time of the second image, and the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the one or more images; then, the terminal determines that the image with the largest stability value among the one or more images is the third image. Based on the stability value of the second image determined in the above manner is greater than that of the third image, so the second image can be considered as the main reference image for generating the target image, and the third image can be considered as the secondary reference image for generating the target image.

In addition, the terminal may also determine the stability value of each of the N images located before and after the first image, and determine the image with the largest stability value among the N images as the second image. That is, the terminal determines the stability values of images within a specific range with the first image as the center, and determines the image with the largest stability value as the second image. Finally, the third image is determined based on the second image.

For example, assuming that the first image is image t, the terminal determines the second image and the third image from images tk to image t-1 and images t+1 to image t+k. In addition, assuming that the stability value of any image p from images tk to image t-1 and images t+1 to image t+k is H, and the jitter value of image p is p _d . Then, the stability value H is specifically: H = 1/(p _d +100*|pt|). It can be seen that the larger the jitter value of image p, the smaller the stability value of image p; the larger the acquisition time difference between image p and the first image, the smaller the stability value of image p.

In this way, the terminal can calculate the stability values H of images t-k to t-1 and images t+1 to t+k respectively, and select an image with the largest stability value H from images t-k to t-1 and images t+1 to t+k as the second image. If the second image is one of images t-k to t-1, the terminal continues to select an image with the largest stability value H from images t+1 to t+k as the third image. If the second image is one of images t+1 to t+k, the terminal continues to select an image with the largest stability value H from images t-k to t-1 as the third image.

Step 304: Generate a target image according to the first image, the second image and the third image.

In this embodiment, the target image is used to replace the first image. After generating the target image, the terminal may replace the first image in the multiple images with the target image to obtain a new multiple images, and output the new multiple images.

Among them, the first image and the third image are used to determine the motion vector corresponding to the image block in the second image, the motion vector is used to represent the relative displacement between the image block in the third image and the image block in the second image, and the target image is obtained based on the image block in the second image and the motion vector.

Specifically, the terminal may divide the second image into a plurality of image blocks, and determine the motion vectors corresponding to the plurality of image blocks respectively according to the first image and the third image, so that each of the plurality of image blocks can find a matching image block in the first image and the third image based on the motion vector corresponding to the image block. That is, for each image block in the second image, the corresponding image block can be found in the first image and the third image based on the motion vector of the image block, and the corresponding image blocks in the first image and the third image are matched with the image block in the second image. That is, the corresponding image block in the first image is similar to the image block in the second image, and the corresponding image block in the third image is also similar to the image block in the second image. Finally, the terminal generates the target image according to the motion vectors corresponding to the plurality of image blocks respectively and the plurality of image blocks.

In this embodiment, the shaking image in the video is located by determining the shaking value of the image in the video. Then, the images before and after the shaking image are used as reference images, and the relative displacement between the image blocks in the reference image is obtained based on the shaking image and the reference image. Finally, a target image for replacing the shaking image is generated based on the image blocks in the reference image and the relative displacement corresponding to the image blocks. Through this solution, the shaking image in the video can be replaced with a normally displayed image, thereby solving the problem of abnormal expansion of the light area in the video screen caused by lens shaking.

In order to facilitate understanding of the above-mentioned process of the terminal determining the motion vector corresponding to the image block in the second image, the motion estimation and motion compensation (MEMC) method in the related art is briefly introduced below.

The MEMC method is a technology currently used for anti-shake or frame rate conversion. The MEMC method estimates the motion trajectory of continuously moving objects in the image and obtains the motion vectors of multiple image blocks in the image. Then, the image blocks and the obtained motion vectors are combined to interpolate the intermediate image, thereby increasing the video frame rate or improving the jitter and tailing problems during video playback.

Please refer to Figure 4, which is a schematic diagram of a MEMC method provided in an embodiment of the present application. As shown in Figure 4, Image 1 and Image 2 are two consecutive images, Image 1 is the image collected first, Image 2 is the image collected later, and Image 3 is an image obtained by interpolation based on Image 1 and Image 2.

During the process of executing MEMC in the terminal, the terminal performs block operations on image 1 and image 2 respectively, and obtains multiple image blocks in image 1 and multiple image blocks in image 2, and the size of each image block is generally 8*8 pixels or 16*16 pixels. For each image block in image 1, the terminal searches for the image block in image 2 that is most similar to the image block in image 1, that is, searches for the image block in image 2 that matches the image block in image 1. For example, as shown in FIG4, image block B in image 2 is most similar to image block A in image 1, so it can be determined that image block B matches image block A. Based on image block B and image block A, the motion vector corresponding to image block A can be determined, and the motion vector is the relative displacement between image block B and image block A. In other words, after image block A moves based on the motion vector, image block A can move to the position where image block B is located. In this way, after determining the image block that matches each image block in image 1 in image 2, the motion vector corresponding to each image block in image 1 can be obtained. The above process of obtaining the motion vector corresponding to the image block is the motion estimation part in the MEMC method.

After obtaining the motion vector corresponding to each image block in image 1, the terminal can move each image block in image 1 according to half of the corresponding motion vector based on the motion vector corresponding to each image block, and finally obtain image 3. For example, as shown in Figure 4, after image block A in image 1 is moved according to half of the motion vector, the image block moves to position P in image 3, and position P is actually located in the middle of the position of image block A and the position of image block B. The above process of generating images based on motion vectors is the motion compensation part in the MEMC method.

However, the above-mentioned MEMC method also has its shortcomings. Since the MEMC method searches for matching image blocks based on the most similar principle, when there are multiple image blocks with close similarity in the searched image, it is easy to obtain an erroneous motion vector, which leads to obvious errors in the final generated image.

Please refer to Figure 5, which is another schematic diagram of a MEMC method provided in an embodiment of the present application. As shown in Figure 5, it is assumed that image 2 also includes image block B', and image block B' is similar to image block A. That is to say, for image block A, image block B and image block B' in image 2 are similar to image block A. In this case, when image 3 is generated based on the motion vector corresponding to image block A, image block A may be moved to position P or position P' in image 3. Assuming that the image block that actually matches image block A in image 2 is image block B, then when image block A finally appears in position P', it will cause obvious errors in image 3.

In view of this, the embodiment of the present application introduces an original image based on the MEMC method of the related art, and further determines the motion vector of the image block based on the original image, thereby ensuring the quality of the generated target image.

Specifically, in this embodiment, in the process of obtaining the motion vector corresponding to the image block in the second image, the terminal jointly determines the motion vector corresponding to the image block in the second image based on the third image and the first image, so that each image block in the second image can find a matching image block in the first image and the third image based on the motion vector corresponding to the image block. In this way, the terminal can ensure that the obtained motion vector is accurate.

Taking Figure 5 as an example, when it is determined that image block B and image block B' in image 2 are similar to image block A, position P and position P' on the original image between image 1 and image 2 are determined based on image block B and image block B'. Then, image block P at position P and image block P' at position P' in the original image are obtained, and the similarity between image block A and image block P and image block P' is calculated. Finally, the motion vector of image A is determined by combining the similarity between image block A and image block B and image block B', as well as the similarity between image block A and image block P and image block P'. In this way, moving image block A based on the final motion vector can move image block A to the correct position P.

For ease of understanding, the process in which the terminal determines motion vectors corresponding to multiple image blocks in the second image respectively according to the first image and the third image in this embodiment will be described in detail below.

In a possible implementation, for multiple image blocks in the second image, the terminal obtains the motion vector corresponding to each of the multiple image blocks one by one. The following takes the process of the terminal obtaining the motion vector corresponding to one of the multiple image blocks as an example to illustrate how the terminal determines the motion vectors corresponding to the multiple image blocks in the second image.

First, the terminal obtains multiple candidate motion vectors corresponding to the first image block. The multiple image blocks in the second image include the first image block, and the first image block can be any one of the multiple image blocks. Optionally, the multiple candidate motion vectors include: one or more preset motion vectors, one or more randomly generated motion vectors and/or motion vectors corresponding to image blocks adjacent to the first image block.

It can be understood that, since the second image and the third image are two images captured at similar times, the displacement of the object in the second image relative to the third image is small, so that it can be considered that the motion vector corresponding to the image block in the second image is within a preset range. In this way, the terminal can obtain one or more preset motion vectors and use these preset motion vectors as candidate motion vectors. In simple terms, it is assumed that the object in the second image is displaced during the acquisition gap between the second image and the third image, but since the acquisition time between the second image and the third image is very close, it can be considered that the displacement of the object in the second image is within a certain range. For example, the preset motion vectors may include (0, -1), (-1, 0), (-1, -1), (-1, 1), (1, -1), (0, 0), (0, 1), (1, 0) and (1, 1), that is, it is considered that the displacement of the image block in the second image is within 1 pixel.

In addition, in order to ensure that the motion vector actually corresponding to the first image block can be obtained as much as possible, one or more motion vectors can be randomly generated or the motion vector corresponding to the image block adjacent to the first image block can be selected as the candidate motion vector, so as to avoid the value range of the candidate motion vector being too limited. Since the first image block and the image block adjacent to the first image block have a greater probability of representing different parts of the same object, the motion vectors corresponding to the first image block and the image block adjacent to the first image block are likely to be the same. Therefore, selecting the motion vector corresponding to the image block adjacent to the first image block as the candidate motion vector can ensure the comprehensiveness of the candidate motion vector. Exemplarily, when the terminal obtains the motion vector corresponding to the image block in the order from left to right and from top to bottom, the image blocks adjacent to the first image block are the four image blocks located at the upper left, upper, upper right and left of the first image block. When the terminal obtains the motion vector corresponding to the image block in other orders, the image block adjacent to the first image block can be an image block located in other positions of the first image block. This embodiment does not specifically limit the position of the image block adjacent to the first image block.

After obtaining multiple candidate motion vectors, the terminal can determine the second image block and the third image block corresponding to each candidate motion vector in the multiple candidate motion vectors according to the position of the first image block, the first image includes the second image block, and the third image includes the third image block. Since the candidate motion vector represents the relative displacement between the image block in the third image and the image block in the second image, the terminal can determine the second image block in the first image and the third image block in the third image based on the position of the first image block in the second image and the candidate motion vector. Exemplarily, please refer to Figure 6, which is a schematic diagram of determining an image block based on a candidate motion vector provided in an embodiment of the present application. As shown in Figure 6, assuming that the second image is the previous image of the first image, the third image is the next image of the first image, the coordinates of the position of the first image block in the second image are (0,0), and the candidate motion vector corresponding to the first image block is (2,-2). Then, based on the candidate motion vector corresponding to the first image block, the coordinates of the position of the third image block in the third image can be obtained as (2,-2), and the coordinates of the position of the second image block in the first image are (1,-1).

Secondly, the terminal determines the target error value corresponding to each candidate motion vector according to the first image block, the second image block and the third image block, and the target error value is obtained based on the error value between the first image block and the second image block and the error value between the first image block and the third image block. Among them, the error value between the two image blocks represents the difference between the two image blocks, and the larger the error value, the larger the difference between the two image blocks. For example, after determining the second image block and the third image block corresponding to the first image block based on the candidate motion vector, the terminal can obtain the error value between the first image block and the second image block and the error value between the first image block and the third image block based on the Sum of Absolute Difference (SAD) algorithm. Then, the terminal determines the target error value based on the error value between the first image block and the second image block and the error value between the first image block and the third image block. For example, the error value between the first image block and the second image block and the error value between the first image block and the third image block are added to obtain the target error value.

Finally, the terminal determines, according to the target error value corresponding to each candidate motion vector, a motion vector with the smallest target error value among the multiple candidate motion vectors as the motion vector corresponding to the first image block.

The above is the process of the terminal obtaining the motion vector of the first image block in the second image. In actual applications, the terminal can obtain the motion vector corresponding to each image block in the second image based on the above process, and finally obtain the motion vectors corresponding to multiple image blocks in the second image.

In practical applications, since the second image includes a large number of image blocks, some of the image blocks in the second image may not be able to determine the corresponding motion vectors because they cannot find matching image blocks in the third image. Alternatively, some of the image blocks in the second image may move to the same position after being moved based on the corresponding motion vectors. In this way, in the process of generating a target image based on multiple image blocks in the second image, there may be areas in the generated target image where images cannot be generated based on the multiple image blocks. These areas where images cannot be generated can be called hole areas.

Exemplarily, please refer to Figure 7, which is a schematic diagram of generating a target image provided by an embodiment of the present application. As shown in Figure 7, the image block at position (1, 1) in the second image shows a person, the image block at position (2, 1) in the second image shows a roadblock, and the image block at position (3, 1) in the second image shows a box. The image block at position (1, 1) in the first image shows a fire hydrant, the image block at position (2, 1) in the first image shows a person, and the image block at position (3, 1) in the first image shows a box. The image block at position (1, 1) in the third image shows a fire hydrant, the image block at position (2, 1) in the third image shows a roadblock, and the image block at position (3, 1) in the second image shows a person.

In FIG7 , when the motion vectors corresponding to the image blocks in the second image are obtained based on the first image and the third image, the motion vector corresponding to the image block 1 at position (1, 1) in the second image is (2, 0), the motion vector corresponding to the image block 2 at position (2, 1) in the second image is (0, 0), and the motion vector corresponding to the image block 3 at position (3, 1) in the second image is (0, 0). Based on the motion vectors obtained above, the image block 1 in the second image moves to position (2, 1) in the target image, the image block 2 also moves to position (2, 1) in the target image, and the image block 3 moves to position (3, 1) in the target image. Finally, there is no image that can be displayed at position (1, 1) in the target image, that is, position (1, 1) is a hole area.

In this embodiment, after obtaining the motion vectors corresponding to the multiple image blocks in the second image, the terminal can determine the corresponding position of the image block in the target image based on the motion vector corresponding to each image block. If different image blocks in the second image have the same position in the target image, the terminal needs to determine an image block displayed on the target image from the multiple different image blocks. For example, in FIG7 , both image block 1 and image block 2 in the second image need to be moved to position (2, 1) in the target image, that is, the corresponding positions of image block 1 and image block 2 in the target image are both (2, 1).

Specifically, after the terminal determines that the corresponding positions of multiple different image blocks in the second image are the same in the target image, the terminal respectively obtains the target error values of the motion vectors corresponding to the multiple different image blocks. It can be seen from the above embodiment that the target error value of the motion vector of each image block in the second image is obtained based on the error value between the image block and the image block at the corresponding position in the first image and the error value between the image block and the image block at the corresponding position in the third image. Finally, the terminal determines an image block with the smallest target error value among the multiple different image blocks, and displays the image block with the smallest target error value on the target image.

Exemplarily, for the image block 1 and the image block 2 in FIG. 7 , the target error value of the motion vector corresponding to the image block 1 is obtained based on the error value between the image block 1 and the image block at position (2,1) in the first image and the image block 1 and the image block at position (3,1) in the third image; the target error value of the motion vector corresponding to the image block 2 is obtained based on the error value between the image block 2 and the image block at position (2,1) in the first image and the image block 2 and the image block at position (3,1) in the third image. Obviously, the target error value of the motion vector corresponding to the image block 1 is smaller than the target error value of the motion vector corresponding to the image block 2. Therefore, the image block 1 in the second image at position (2,1) in the target image is finally displayed.

In order to eliminate the hole areas in the target image, in this embodiment, motion vectors corresponding to multiple image blocks in the third image are determined based on the first image and the second image, and the target image is updated according to the motion vectors corresponding to the multiple image blocks in the third image, and finally an updated target image is obtained.

Exemplarily, after the terminal generates a target image based on multiple image blocks of the second image, if there are areas in the target image where an image cannot be generated based on the multiple image blocks, the terminal divides the third image to obtain multiple image blocks of the third image.

Then, the terminal determines the motion vectors corresponding to the multiple image blocks of the third image respectively according to the first image and the second image. The process of the terminal determining the motion vectors corresponding to the multiple image blocks of the third image respectively is similar to the process of the terminal determining the motion vectors corresponding to the multiple image blocks of the second image. For details, please refer to the description of the above embodiment, which will not be repeated here.

Finally, the terminal updates the target image according to the motion vectors corresponding to the multiple image blocks of the third image and the multiple image blocks of the third image, and obtains a new target image, and the new target image is used to replace the first image. The terminal updates the target image means that the terminal moves the image blocks in the third image to the hole area in the target image according to the motion vectors corresponding to the multiple image blocks of the third image, thereby filling the hole area. For areas in the target image where images already exist, these areas where images already exist are no longer updated.

For example, please refer to Figure 8, which is a schematic diagram of another method for generating a target image provided by an embodiment of the present application. Figure 8 is based on the embodiment shown in Figure 7, and updates the target image based on the motion vector corresponding to the image block in the third image to obtain an updated target image.

In Figure 8, when the motion vector corresponding to the image block in the second image is obtained based on the first image and the third image, the motion vector corresponding to the image block 1' at the position (1, 1) in the third image is (0, 0), the motion vector corresponding to the image block 2' at the position (2, 1) in the third image is (0, 0), and the motion vector corresponding to the image block 3' at the position (3, 1) in the third image is (-2, 0).

Based on the motion vector obtained above, the image block 1' in the second image needs to be moved to the position (1, 1) in the target image, the image block 2' needs to be moved to the position (2, 1) in the target image, and the image block 3' needs to be moved to the position (2, 1) in the target image. Since the position of the hole area in the generated target image is (1, 1), it is only necessary to move the image block 1' in the second image to the position (1, 1) in the target image based on the motion vector to complete the update of the target image and obtain the updated target image. As can be seen from Figure 8, the updated target image completes the filling of the hole area, and the updated target image can stably display the content included in the first image.

For ease of understanding, the video processing method of this embodiment will be described in detail below with reference to specific examples.

Please refer to Figure 9, which is a schematic diagram of the application architecture of a video processing method provided in an embodiment of the present application. As shown in Figure 9, the terminal includes an image acquisition device, a gyroscope, an image signal processing (Image Signal Processing, ISP) module and a display component. The ISP module includes a tremor judgment unit and a de-tremor unit. Optionally, the ISP module also includes an EIS unit. Among them, the image acquisition device transmits the multiple images acquired to the EIS unit, and the EIS unit performs an anti-shake operation on the multiple images. The tremor judgment unit obtains the multiple images output by the EIS unit, and determines whether there is a tremor image in the multiple images, and finally the de-tremor unit generates a target image for replacing the tremor image. After the de-tremor unit replaces the tremor image in the multiple images with the target image, the multiple images after the target image is replaced are output to the display component, and the display component displays the multiple images after the target image is replaced.

Please refer to Figures 10 and 11, Figure 10 is a schematic diagram of a process of replacing a trembling image provided by an embodiment of the present application, and Figure 11 is another schematic diagram of a process of replacing a trembling image provided by an embodiment of the present application. The video processing method shown in Figure 10 includes the following steps 1001 to 1006.

Step 1001, acquiring multiple images, and determining whether image t among the multiple images has a local light area.

After the image acquisition device of the terminal acquires the image, the shake determination unit may acquire multiple images, which may be anti-shake processed by the EIS unit. The shake determination unit determines whether the images in the multiple images are shaken images one by one.

Exemplarily, the tremor determination unit first determines whether there is a local light area in image t among the multiple images. Taking the YUV format of the color space of the image as an example, the terminal can first determine the number of target pixels in the light area in image t, and then the terminal determines the ratio of the target pixels to the total number of pixels in image t. If the ratio of the target pixels to the total number of pixels in the image is within 2%-20%, it is determined that there is a local light area in the image. In the specific implementation process of the terminal, the terminal can set the initial values of the variables total and light to be 0. For each pixel p in image t, the value of the variable total is increased by 1; if the pixel p is located in the light area, light is also increased by 1. After all pixels in image t are traversed, r=light/total is calculated. If r is within 2%-20%, it is considered that there is a local light area in image t; if r is not within 2%-20%, it is considered that there is no local light area in image t, that is, image t is not a tremor image.

Step 1002: determine whether the jitter value of the image t is greater than or equal to a second preset threshold.

After determining that there is a local light area in the image t, the tremor determination unit further determines whether the jitter value of the image t is greater than or equal to a second preset threshold value to determine whether the image t is a tremor image. The second preset threshold value may be 360, for example.

If the jitter value of image t is greater than or equal to the second preset threshold, it can be determined that image t is a trembling image; if the jitter value of image t is less than the second preset threshold, it can be determined that image t is not a trembling image.

In this embodiment, the shake determination unit may obtain gyroscope data collected by a gyroscope in the terminal, and determine the shake value of the image t based on the gyroscope data.

First, the corresponding gyroscope data is obtained according to the acquisition timestamp information of the image t. If the sampling rate f of the gyroscope data is less than the preset threshold value fs, the gyroscope data is upsampled based on the upsampling algorithm so that the gyroscope data can be greater than or equal to the threshold value fs. Among them, the threshold value fs can be 1000Hz, for example, and the threshold value fs can also be adjusted as needed in practical applications. The upsampling algorithm used to upsample the gyroscope data can adopt any existing sampling algorithm, such as the cubic convolution interpolation algorithm. Simply put, if f<fs, a new sampling rate f'≥fs is selected, and f' is, for example, an integer multiple of f; then, the upsampling algorithm is executed to increase the sampling rate f to f'.

The gyroscope data collects angular velocities in three dimensions, and the angular velocities collected by the gyroscope in different dimensions are respectively represented by x, y, and z. Specifically, an example of gyroscope data can be found in Table 1.

Table 1

serial number time x y z 0(start) t0 - - - 1 t1 x1 y1 z1 2 t2 x2 y2 z2 3 t3 x3 y3 z3 … … … … … 34(end) t4 x34 y34 z34

In Table 1, start refers to the time when the current image starts to be exposed, and end refers to the time when the current image ends to be exposed.

Since the data x1, y1, z1, etc. represent angular velocity, the angular displacement from time t[i] to t[i+1] can be approximately calculated as s[i] = ((t[i+1]-t[i])*x[i], (t[i+1]-t[i])*y[i], (t[i+1]-t[i])*z[i]). This formula includes the angular displacement from t33 to t34. In particular, the angular displacement from t0 to t1 can be approximated using the gyroscope data numbered 1, i.e., s[0] = ((t[1]–t[0])*x[1], (t[1]-t[0])*y[1], (t[1]-t[0])*z[1]).

Based on the angular displacement s[i], the gyroscope positions at times t1, t2, ..., t34 can be accumulated and calculated. Specifically, the t0 position l[0] is (0,0,0); Therefore, based on the above gyroscope data, 35 position data of the gyroscope in each dimension can be obtained.

For the 35 position data on the x-axis, take the position data with the largest value and the position data with the smallest value among the 35 position data, subtract them and then square them to get the x-axis result. Similarly, based on the 35 position data on the y-axis and the 35 position data on the z-axis, the y-axis result and the z-axis result are obtained. Finally, the sum of the x-axis result, the y-axis result and the z-axis result is calculated, and the square root of the sum is calculated to get the image jitter value.

In addition, in addition to obtaining the x-axis result by calculating the difference between the position data with the largest value and the position data with the smallest value among the 35 position data, the x-axis result can also be obtained by calculating the variance between the 35 position data. Similarly, the y-axis result and the z-axis result are obtained by calculating the variance between the 35 position data on the y-axis and the variance between the 35 position data on the z-axis. Finally, the sum of the x-axis result, the y-axis result and the z-axis result is calculated, and the square root of the sum is calculated to obtain the image jitter value.

Optionally, in some cases, when the terminal experiences low-frequency jitter, the image captured by the terminal will be blurred, but the phenomenon of abnormal expansion of the light area may not appear in the image; when the terminal experiences high-frequency and large-amplitude jitter, the image captured by the terminal will show abnormal expansion of the light area. Based on this, in the process of obtaining the jitter value of the image based on the gyroscope data, the spectrum information corresponding to the gyroscope data can also be obtained, and the jitter value of the image can be determined based on the spectrum information.

For example, for the data in Table 1, a Fourier transform is performed on the dimensions of the x-axis, the y-axis, and the z-axis to obtain a spectrum diagram. For example, the Fourier transform result on the x-axis can be obtained based on the following formula 1.

Wherein, _Xk is the Fourier transform result, k={0,1,2,3…,34}, x[n] represents the gyroscope data on the x-axis, and N is 35.

Assume w = |X ₀ |/sigma(|X _i |), w represents the proportion of the lowest frequency information in the x-axis gyroscope data, and sigma() represents the summation operation. After obtaining the above x-axis result, multiply the x-axis result by (1-w) to obtain the updated x-axis result. Similarly, the updated y-axis and z-axis results are obtained in a similar manner. Finally, the sum of the updated x-axis result, y-axis result, and z-axis result is obtained, and the square root of the sum is obtained to obtain the image jitter value.

Step 1003, obtaining the jitter value from image t-k to image t+k.

When determining that image t is a trembling image, the trembling judgment unit sends image t to the de-trembling unit. In addition, the trembling judgment unit obtains the jitter values of image t-k to image t+k, and inputs the jitter values of image t-k to image t+k to the de-trembling unit. The jitter values of image t-k to image t may be obtained before obtaining the jitter value of image t.

In this embodiment, the manner in which the shake determination unit acquires the shake values of the image t-k to the image t+k is similar to the manner in which the shake value of the image t is acquired, and will not be described in detail herein.

Step 1004: Determine a primary reference image M and a secondary reference image m according to the jitter values from image t-k to image t+k.

After obtaining the jitter values of images tk to t+k, the terminal determines the primary reference image M and the secondary reference image m in images tk to t-1 and images t+1 to t+k. In addition, it is assumed that the stability value of any image p from images tk to t-1 and images t+1 to t+k is H, and the jitter value of image p is p _d . Then, the stability value H is specifically: H=1/(p _d +100*|pt|). It can be seen that the larger the jitter value of image p, the smaller the stability value of image p; the farther the image p is from image t, the smaller the stability value of image p.

In this way, the terminal can calculate the stability values H of images t-k to t-1 and images t+1 to t+k respectively, and select an image with the largest stability value H from images t-k to t-1 and images t+1 to t+k as the main reference image M. If the main reference image M is an image from images t-k to t-1, the terminal continues to select an image with the largest stability value H from images t+1 to t+k as the secondary reference image m. If the main reference image M is an image from images t+1 to t+k, the terminal continues to select an image with the largest stability value H from images t-k to t-1 as the secondary reference image m.

In addition, the de-shaking unit may also define an algorithm help value Q, which is positively correlated with the stability value H1 of the main reference image M and the stability value H2 of the secondary reference image m. The algorithm help value Q represents the help value for generating a target image with stable display based on the main reference image M and the secondary reference image m. For example, Q = (2*H1+H2)/3 may be defined. On this basis, a help value threshold may be set in advance. If the algorithm help value Q is lower than the help value threshold, it may be determined that it is difficult to make the image t more stable based on the main reference image and the secondary reference image. For example, when the main reference image and the secondary reference image near the image t are also shaky images, the main reference image and the secondary reference image cannot make the image t more stable. Therefore, when the algorithm help value Q is lower than the help value threshold, the de-shaking unit may end the processing of the image t, that is, no longer generate a target image for replacing the image t.

Step 1005 , generating a target image according to the primary reference image M, the secondary reference image m and the image t.

In this embodiment, the target image for replacing the image t is generated by dividing the primary reference image M and the secondary reference image m into a plurality of image blocks. Since when the primary reference image M and the secondary reference image m have high resolutions, dividing the primary reference image M and the secondary reference image m into image blocks and obtaining motion vectors of the image blocks requires excessive computing power, a pyramid downsampling method may be used in this embodiment to obtain motion vectors of image blocks in the primary reference image M and the secondary reference image m.

Specifically, firstly, pyramid downsampling is performed on the main reference image M and the secondary reference image m to obtain multi-layer downsampled images corresponding to the main reference image M and the secondary reference image m. For example, when the resolution of the main reference image M is 1920*1080, the main reference image M can be downsampled 4 times to obtain four layers of downsampled images corresponding to the main reference image M. The resolutions of the four layers of downsampled images can be, for example, 480*270, 120*68, 60*34, and 30*17. For the main reference image M and the secondary reference image m, starting from the layer with the smallest resolution to the layer where the original image is located, the motion vector of the image block of the main reference image Mx of each layer is obtained. Among them, the main reference image Mx represents the image corresponding to the main reference image M at any layer.

First, for a main reference image Mx of the current layer, the main reference image Mx is divided according to a specific size (for example, 8*8 pixels) to obtain a plurality of image blocks in the main reference image Mx.

If the main reference image Mx is an image of a layer with the smallest resolution, then for the image block q in the main reference image Mx, the candidate motion vector of the image block q can be determined to be a preset candidate motion vector. For example, the preset motion vector may include (0, -1), (-1, 0), (-1, -1), (-1, 1), (1, -1), (0, 0), (0, 1), (1, 0), and (1, 1). If the main reference image Mx is not an image of a layer with the smallest resolution, then for the image block q in the main reference image Mx, the candidate motion vector of the image block q can be determined to include: the motion vector of the image block corresponding to the image block q in the previous layer of the image, the motion vector of the image block adjacent to the image block q, and a randomly generated motion vector.

For all candidate motion vectors of image block q, calculate the position of image block q mapped to the secondary reference image mx and image t according to the candidate motion vector. Specifically, when image block q is mapped to image t according to the candidate motion vector vec, the candidate motion vector needs to be scaled to vec’=((t-M)/(m-M))*vec according to the values of M, m, and t. After obtaining the image block q1 corresponding to image block q in the secondary reference image mx and the image block q2 corresponding to image block q in image t, calculate the error values between image block q and image block q1 and image block q2 respectively. Exemplarily, the error values between image blocks can be calculated based on a variety of algorithms, such as the SAD algorithm. Finally, assuming that the error value between image block q and image block q1 is SAD1, and the error value between image block q and image block q2 is SAD2, the error value D of the candidate motion vector is obtained by weighted summing of the two error values. Among them, the weights of the two error values can be set arbitrarily, for example, D=SAD1+αSAD2, α=1/2.

After obtaining the error values D corresponding to all candidate motion vectors of the image block q, the candidate motion vector with the smallest error value D is selected as the motion vector of the image block q in the current pyramid layer. Finally, by repeating the above process, the motion vectors of all image blocks in the main reference image Mx can be calculated.

In addition, after obtaining the motion vector of the layer where the original image of the main reference image M is located, the motion vector of the main reference image M can be smoothed and corrected to eliminate individual obviously unreasonable motion vectors. For example, for a certain image block in the main reference image, if the motion vector of the image block is too different from the motion vectors of other image blocks adjacent to the image block, the motion vector of the image block is eliminated.

After obtaining the motion vector of each image block in the main reference image M, assuming that the target image to be generated is image t’, the motion vector of each image block in the main reference image M can be converted into a motion vector vec’=(-(t-M)/(m-M))*vec pointing from image t’ to the main reference image M according to the conversion ratio.

In this way, if there is an image block in the image t' that does not have a motion vector pointing to the main reference image M, it means that the image block belongs to a hole area, so the image block can be marked as a hole area.

After marking the hole area in the image t’, the image blocks in the non-hole area in the image t’ can be filled according to the image blocks in the main reference image M to obtain the image t’ still including the hole area.

It is worth noting that in the process of filling the image blocks in the non-hole area of image t’ according to the image blocks in the main reference image M, if there are multiple different image blocks in the main reference image M pointing to the same image block in image t’, the terminal selects the image block with the smallest error value D from the multiple different image blocks according to the error value D of the motion vectors of the multiple different image blocks, and fills the image block with the smallest error value D into image t’.

In addition, the de-shaking unit can obtain the motion vector corresponding to each image block in the secondary reference image m, and fill the hole area in the image t' according to the image blocks in the secondary reference image m to obtain the final target image t'.

Step 1006, replacing image t with the target image.

Finally, after the target image is obtained based on the primary reference image M, the secondary reference image m and the image t, the image t is replaced with the target image.

Similarly, for multiple images captured by the terminal, based on the above steps 1001 to 1006, the tremor images in the multiple images can be determined, and the tremor images in the multiple images can be replaced with the generated stable images, and finally the multiple images after the stable images are replaced are output.

The above embodiments introduce the process of processing a video by a terminal during video shooting. The following will introduce the process of processing a video when the terminal is in night scene shooting mode. Please refer to Figure 12, which is a flow chart of another video processing method provided by this embodiment, and the video processing method includes the following process.

Step 1201: When the terminal is in a night scene shooting mode, detect whether there is a light area in the image shot by the terminal.

The terminal can determine whether there is a light area in the image by detecting the brightness values of pixels in the captured image. Specifically, the brightness values of the pixels in the light area are all greater than a specific threshold, and the difference between the brightness values of the pixels in the light area and the brightness values of the pixels in the adjacent area of the light area is greater than a preset difference. The value of the above-mentioned specific threshold can be 180-230, for example, the specific threshold is specifically 224; the value of the preset difference can be 25-50, for example, the preset difference is 32. For example, when the terminal detects that the brightness values of the pixels in a certain area in the image are all greater than 224, and the difference between the brightness values of the pixels in the area and the brightness values of the pixels in the adjacent area is greater than 32, the terminal can determine that the area is a light area. The adjacent area of the light area can be composed of pixels adjacent to the pixels in the light area, that is, the pixels in the adjacent area of the light area are adjacent to the pixels in the light area.

In addition, after the terminal determines that there is a light area in the image, the terminal can also determine the position of the light area in the image according to the coordinates of the pixels located in the light area. The position of the light area in the image can be determined by the coordinates of the pixels at the boundary of the light area, that is, the area surrounded by the pixels at the boundary of the light area is the light area. The terminal can determine the position of the light area by recording the coordinates of the pixels at the boundary of the light area or the coordinates of all pixels in the light area.

Step 1202: When there is a light area in the image captured by the terminal, the terminal determines a first image in the acquired image sequence, and an area adjacent to the light area in the first image is overexposed.

The overexposure of the adjacent area of the light area in the first image means that the brightness value of the pixels in the adjacent area is too high. The reason why the adjacent area of the light area in the first image is overexposed is that the terminal has a jitter displacement when capturing the first image, so that the photosensitive element corresponding to the adjacent area in the terminal also senses the light in the light area, which ultimately causes the adjacent area of the light area in the first image to be overexposed.

Step 1203: The terminal determines a second image and a third image in the image sequence according to the first image.

The acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image, and the overexposure degree of the adjacent area of the light area in the second image and the third image is less than the overexposure degree of the adjacent area of the light area in the first image. The light areas in the second image and the third image correspond to the light areas in the first image, that is, the light areas in the second image and the third image represent the same picture content as the light areas in the first image. For example, the light areas in the second image and the third image represent the luminous area of a street lamp; the light area in the first image also represents the luminous area of the same street lamp.

Specifically, the overexposure degree of the adjacent areas of the light area in the second image and the third image is smaller than the overexposure degree of the adjacent areas of the light area in the first image, which may mean: the area of the overexposed area in the adjacent areas of the light area in the second image and the third image is smaller than the area of the overexposed area in the adjacent areas of the light area in the first image; or, the brightness value of the overexposed pixel in the adjacent areas of the light area in the second image and the third image is smaller than the brightness value of the overexposed pixel in the adjacent areas of the light area in the first image. Optionally, the area of the overexposed area in the adjacent areas of the light area in the second image and the third image may be 0, that is, the adjacent areas of the light area in the second image and the third image are not overexposed. In general, compared with the first image, the second image and the third image are collected by the terminal under the condition of small jitter displacement or no jitter displacement, so the overexposure degree of the adjacent areas of the light area in the second image and the third image is smaller than the overexposure degree of the adjacent areas of the light area in the first image.

Step 1204: The terminal generates a target image based on the first image, the second image, and the third image, the target image being used to replace the first image, and the overexposure degree of the area adjacent to the light area of the target image being smaller than the overexposure degree of the area adjacent to the light area of the first image. The second image and the third image are used as reference images to generate a target image that replaces the first image, so that the overexposure degree of the area adjacent to the light area of the generated target image is smaller than the overexposure degree of the area adjacent to the light area of the first image.

Step 1205: The terminal generates a video based on the target image. Specifically, the terminal may replace the first image in the acquired image sequence with the target image, thereby generating a video.

In this embodiment, by locating the overexposed trembling image in the vicinity of the light area in the night scene shooting mode, the images before and after the trembling image are used as reference images, and a target image for replacing the trembling image is generated based on the trembling image and the reference image. Through this solution, the trembling image in the video can be replaced with a normally displayed image, thereby solving the problem of abnormal expansion of the bright part of the light area in the video screen caused by lens shaking.

Optionally, the method provided in this embodiment also includes: when the illumination of the ambient light is less than a first preset threshold, the terminal enters a night scene shooting mode. Exemplarily, the value of the first preset threshold can be 30-60, for example, the first preset threshold is 50. During the shooting process, the terminal can detect the illumination of the ambient light, and when the illumination of the ambient light is less than 50, the terminal enters the night scene shooting mode. Or, when the terminal obtains an instruction for triggering entry into the night scene shooting mode, the terminal enters the night scene shooting mode. For example, in the process of the user operating the terminal to shoot, the user can send an instruction to the terminal to enter the night scene shooting mode by touching the terminal screen; in this way, when the terminal obtains an instruction for triggering entry into the night scene shooting mode, the terminal enters the night scene shooting mode.

In addition, the terminal can also determine whether the current shooting scene is a night scene according to the picture content of the preview image during shooting and/or the ambient brightness value of each area of the preview image, thereby determining whether to enter the night scene shooting mode. For example, when the picture content of the preview image includes the night sky or night scene light source, the terminal can determine that the current shooting scene is a night scene, thereby entering the night scene shooting mode; or, when the ambient brightness value of each area of the preview image meets the brightness distribution characteristics of the image in the night scene environment, the terminal can determine that the current shooting scene is a night scene, thereby entering the night scene shooting mode.

Optionally, the jitter displacement when the terminal acquires the first image is greater than a second preset threshold. The second preset threshold may be 300-400, for example, the second preset threshold is specifically 360. The jitter displacement when the terminal acquires the first image may be referred to as a jitter value of the first image. The terminal may determine the first image in the image sequence by determining whether the jitter value of the image in the image sequence is greater than or equal to the second preset threshold.

In some possible implementations, the terminal may obtain sensor data that records the movement of the terminal when capturing the first image, and calculate the jitter value of the first image based on the sensor data. The terminal may also calculate the contrast value of the image, and determine the jitter value of the image based on the contrast value of the image, that is, the smaller the contrast value of the image, the greater the jitter value of the image. In addition, the terminal may also determine the jitter value of the image based on a pre-trained neural network. Specifically, the manner in which the terminal determines the jitter value of the first image can refer to the above-mentioned embodiment, which will not be repeated here.

In a possible implementation, the terminal may also determine whether there is a local light area in the multiple images. Only when a certain image has a local light area and the jitter value of the image is greater than or equal to a second preset threshold value, the terminal determines the image as a trembling image. Wherein, there is a local light area in the first image, and the ratio of the number of target pixels in the first image to the total number of pixels in the first image is within a preset range, and the target pixel is a pixel in the light area in the first image. Exemplarily, the terminal may determine the number of target pixels in each of the multiple images, and the target pixel refers to a pixel in the light area, wherein the definition of the light area may refer to the above description. Then, the terminal obtains the ratio of the number of target pixels in each image to the total number of pixels in the image, and determines whether the ratio is within a preset range. If the ratio of the number of target pixels in the image to the total number of pixels in the image is within the preset range, the terminal determines that there is a local light area in the image; if the ratio of the number of target pixels in the image to the total number of pixels in the image is not within the preset range, the terminal determines that there is no local light area in the image.

In addition, the manner in which the terminal determines the second image and the third image in the image sequence based on the first image is also similar to the above-mentioned embodiment, and will not be described in detail herein.

On the basis of the embodiments corresponding to FIG. 1a to FIG. 12 , in order to better implement the above-mentioned solutions of the embodiments of the present application, related devices for implementing the above-mentioned solutions are also provided below.

For details, please refer to FIG. 13 , which is a schematic diagram of the structure of a terminal 1300 provided in an embodiment of the present application. The terminal 1300 includes:

The detection unit 1301 is used to detect whether there is a light area in the image captured by the terminal when the terminal is in a night scene shooting mode;

A first determining unit 1302 is configured to, when there is a light area in the image captured by the terminal, determine a first image in the image sequence collected by the terminal, in which an area adjacent to the light area in the first image is overexposed;

A second determining unit 1303 is configured to determine a second image and a third image in the image sequence according to the first image, wherein a capture time of the first image is between a capture time of the second image and a capture time of the third image, and an overexposure degree of an area adjacent to the light area in the second image and the third image is less than an overexposure degree of an area adjacent to the light area in the first image;

The image generating unit 1304 is further configured to generate a target image according to the first image, the second image, and the third image, wherein the target image is used to replace the first image, and the overexposure degree of the neighboring area of the light area of the target image is smaller than the overexposure degree of the neighboring area of the light area of the first image;

The video generating unit 1305 is further configured to generate a video based on the target image.

In a possible implementation, it further includes: a control unit 1306;

The control unit 1306 is used to: when the illumination of the ambient light is less than a first preset threshold, control the terminal to enter the night scene shooting mode; or, when an instruction for triggering entering the night scene shooting mode is obtained, control the terminal to enter the night scene shooting mode.

In a possible implementation manner, a jitter displacement when the terminal captures the first image is greater than a second preset threshold.

In a possible implementation manner, a ratio of the number of target pixels in the first image to the total number of pixels in the first image is within a preset range, and the target pixels are pixels in a light area in the first image.

In a possible implementation, the second determination unit 1303 is used to: respectively determine a stability value of each of the multiple images, where the stability value is the reciprocal of the sum of a jitter value of the image and an acquisition time difference between the image and the first image; determine the image with the largest stability value among the multiple images as the second image; and determine the third image among the multiple images based on the acquisition time of the second image.

In a possible implementation, the second determination unit 1303 is used to: determine one or more images from the multiple images based on the acquisition time of the second image, the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the one or more images; and determine the image with the largest stability value from the one or more images as the third image.

In a possible implementation, the image generation unit 1304 is used to: divide the second image into multiple image blocks; determine the motion vectors corresponding to the multiple image blocks respectively according to the first image and the third image; and generate the target image according to the motion vectors corresponding to the multiple image blocks respectively and the multiple image blocks.

In a possible implementation, the image generation unit 1304 is used to: obtain multiple candidate motion vectors corresponding to a first image block, the multiple image blocks including the first image block; determine, according to the position of the first image block, a second image block and a third image block corresponding to each of the multiple candidate motion vectors, the first image including the second image block, and the third image including the third image block; determine, according to the first image block, the second image block, and the third image block, a target error value corresponding to each candidate motion vector, the target error value being obtained based on an error value between the first image block and the second image block and an error value between the first image block and the third image block; and, according to the target error value corresponding to each candidate motion vector, determine, among the multiple candidate motion vectors, a motion vector with a minimum target error value as the motion vector corresponding to the first image block.

In a possible implementation manner, the multiple candidate motion vectors include: one or more preset motion vectors, one or more randomly generated motion vectors and/or motion vectors corresponding to an image block adjacent to the first image block.

In a possible implementation, the image generation unit 1304 is used to: if there is an area in the target image where an image cannot be generated based on the multiple image blocks, divide the third image to obtain multiple image blocks of the third image; determine the motion vectors corresponding to the multiple image blocks of the third image according to the first image and the second image; update the target image according to the motion vectors corresponding to the multiple image blocks of the third image and the multiple image blocks of the third image to obtain a new target image, and the new target image is used to replace the first image.

The video processing method provided in the embodiment of the present application can be specifically executed by a chip in a terminal, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input/output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the server executes the video processing method described in the embodiments shown in Figures 1a to 11 above. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

Referring to Figure 14, the present application also provides a computer program product. In some embodiments, the method disclosed in Figure 3 above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or encoded on other non-transitory media or products.

14 schematically illustrates a conceptual partial view of an example computer program product including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein.

In one embodiment, computer program product 1400 is provided using signal bearing medium 1401. Signal bearing medium 1401 may include one or more program instructions 1402, which when executed by one or more processors may provide the functionality or portions of the functionality described above with respect to FIG. 2. Thus, for example, with reference to the embodiment shown in FIG. 3, one or more features of steps 301-306 may be undertaken by one or more instructions associated with signal bearing medium 1401. In addition, program instructions 1402 in FIG. 14 also describe example instructions.

In some examples, the signal bearing medium 1401 may include a computer readable medium 1403 such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a ROM or RAM, and the like.

In some embodiments, the signal bearing medium 1401 may include a computer recordable medium 1404, such as, but not limited to, a memory, a read/write (R/W) CD, a R/W DVD, etc. In some embodiments, the signal bearing medium 1401 may include a communication medium 1405, such as, but not limited to, a digital and/or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 1401 may be communicated by a wireless form of the communication medium 1405 (e.g., a wireless communication medium that complies with the IEEE 802.14 standard or other transmission protocol).

The one or more program instructions 1402 may be, for example, computer executable instructions or logic implementation instructions. In some examples, the computing device of the computing device may be configured to provide various operations, functions, or actions in response to the program instructions 1402 communicated to the computing device via one or more of the computer readable medium 1403, the computer recordable medium 1404, and/or the communication medium 1405.

It should be understood that the arrangement described here is only for illustrative purposes. Thus, it will be appreciated by those skilled in the art that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements can be omitted together according to the desired result. In addition, many of the described elements can be implemented as discrete or distributed components, or in any appropriate combination and position to combine other components to implement functional entities.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

Claims (24)

1. A video processing method, applied to video capturing, the method comprising:

when a terminal is in a night scene shooting mode, detecting whether a lamplight area exists in an image shot by the terminal;

when a lamplight area exists in an image shot by the terminal, determining a first image in an image sequence acquired by the terminal, wherein the adjacent area of the lamplight area in the first image is overexposed;

Determining a second image and a third image in the image sequence according to the first image, wherein the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the third image, and the overexposure degree of the adjacent areas of the light areas in the second image and the third image is smaller than that of the adjacent areas of the light areas in the first image; wherein, the lamplight areas in the second image and the third image have a corresponding relation with the lamplight areas in the first image;

Generating a target image according to the first image, the second image and the third image, wherein the target image is used for replacing the first image, and the overexposure degree of the adjacent area of the lamplight area of the target image is smaller than that of the adjacent area of the lamplight area of the first image;

based on the target image, a video is generated.

2. The method according to claim 1, wherein the method further comprises:

When the illuminance of the ambient light is smaller than a first preset threshold value, the terminal enters a night scene shooting mode;

or when the terminal acquires an instruction for triggering to enter a night scene shooting mode, the terminal enters the night scene shooting mode.

3. A method according to claim 1 or 2, wherein the jitter displacement of the terminal when acquiring the first image is larger than a second preset threshold.

4. A method according to claim 3, wherein the ratio of the number of target pixels in the first image to the total number of pixels in the first image is within a predetermined range, the target pixels being pixels in a light area in the first image.

5. The method according to any of claims 1-2, wherein said determining a second image and a third image in said sequence of images from said first image comprises:

determining a stable value of each image in the image sequence, wherein the stable value is the inverse of the sum of the jitter value of the image and the acquisition time difference between the image and the first image;

determining an image with the largest stable value in the image sequence as the second image;

and determining the third image in the image sequence according to the acquisition time of the second image.

6. The method of claim 5, wherein determining the third image in the image sequence based on the time of acquisition of the second image comprises:

Determining one or more images in the image sequence according to the acquisition time of the second image, wherein the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the one or more images;

And determining an image with the largest stable value from the one or more images as the third image.

7. The method of any of claims 1-2, wherein the generating a target image from the first image, the second image, and the third image comprises:

Dividing the second image into a plurality of image blocks;

determining motion vectors respectively corresponding to the image blocks according to the first image and the third image;

And generating the target image according to the motion vectors respectively corresponding to the image blocks and the image blocks.

8. The method of claim 7, wherein determining motion vectors for the respective ones of the plurality of image blocks from the first image and the third image comprises:

Acquiring a plurality of candidate motion vectors corresponding to a first image block, wherein the plurality of image blocks comprise the first image block;

Determining a second image block and a third image block corresponding to each candidate motion vector in the plurality of candidate motion vectors according to the position of the first image block, wherein the first image comprises the second image block, and the third image comprises the third image block;

Determining a target error value corresponding to each candidate motion vector according to the first image block, the second image block and the third image block, wherein the target error value is obtained based on an error value between the first image block and the second image block and an error value between the first image block and the third image block;

And determining a motion vector with the minimum target error value from the plurality of candidate motion vectors as the motion vector corresponding to the first image block according to the target error value corresponding to each candidate motion vector.

9. The method of claim 8, wherein the plurality of candidate motion vectors comprises: one or more preset motion vectors, one or more motion vectors generated randomly, and/or motion vectors corresponding to image blocks adjacent to the first image block.

10. The method of claim 7, wherein the method further comprises:

if the target image has an area which cannot generate an image based on the plurality of image blocks, dividing the third image to obtain the plurality of image blocks of the third image;

Determining motion vectors respectively corresponding to a plurality of image blocks of the third image according to the first image and the second image;

And updating the target image according to the motion vectors corresponding to the image blocks of the third image and the image blocks of the third image to obtain a new target image, wherein the new target image is used for replacing the first image.

11. A video processing apparatus comprising a memory and a processor; the memory stores code, the processor being configured to execute the code, when executed, the video processing apparatus performing the method of any of claims 1 to 10.

12. A video processing apparatus, comprising:

the detection unit is used for detecting whether a light area exists in an image shot by the terminal when the terminal is in a night scene shooting mode;

the first determining unit is used for determining a first image in an image sequence acquired by the terminal when a lamplight area exists in the image shot by the terminal, and overexposure occurs in an adjacent area of the lamplight area in the first image;

A second determining unit, configured to determine a second image and a third image in the image sequence according to the first image, where an acquisition time of the first image is located between an acquisition time of the second image and an acquisition time of the third image, and an overexposure degree of a neighboring area of a light area in the second image and the third image is smaller than an overexposure degree of a neighboring area of the light area in the first image; wherein, the lamplight areas in the second image and the third image have a corresponding relation with the lamplight areas in the first image;

the image generation unit is further used for generating a target image according to the first image, the second image and the third image, wherein the target image is used for replacing the first image, and the overexposure degree of the adjacent area of the lamplight area of the target image is smaller than that of the adjacent area of the lamplight area of the first image;

and the video generating unit is also used for generating a video based on the target image.

13. The apparatus as recited in claim 12, further comprising: a control unit;

The control unit is used for: when the illuminance of the ambient light is smaller than a first preset threshold value, controlling the terminal to enter a night scene shooting mode; or when an instruction for triggering to enter the night scene shooting mode is acquired, controlling the terminal to enter the night scene shooting mode.

14. The apparatus according to claim 12 or 13, wherein the jitter displacement of the terminal when acquiring the first image is larger than a second preset threshold.

15. The apparatus of claim 14, wherein a ratio of a number of target pixels in the first image to a total number of pixels in the first image is within a preset range, the target pixels being pixels in a light area in the first image.

16. The apparatus according to any of the claims 12-13, wherein the second determining unit is further configured to: determining a stable value of each image in the image sequence, wherein the stable value is the inverse of the sum of the jitter value of the image and the acquisition time difference between the image and the first image; determining an image with the largest stable value in the image sequence as the second image; and determining the third image in the image sequence according to the acquisition time of the second image.

17. The apparatus of claim 16, wherein the second determining unit is configured to: determining one or more images in the image sequence according to the acquisition time of the second image, wherein the acquisition time of the first image is between the acquisition time of the second image and the acquisition time of the one or more images; and determining an image with the largest stable value from the one or more images as the third image.

18. The apparatus according to any one of claims 12-13, wherein the image generation unit is configured to: dividing the second image into a plurality of image blocks; determining motion vectors respectively corresponding to the image blocks according to the first image and the third image; and generating the target image according to the motion vectors respectively corresponding to the image blocks and the image blocks.

19. The apparatus of claim 18, wherein the image generation unit is configured to: acquiring a plurality of candidate motion vectors corresponding to a first image block, wherein the plurality of image blocks comprise the first image block; determining a second image block and a third image block corresponding to each candidate motion vector in the plurality of candidate motion vectors according to the position of the first image block, wherein the first image comprises the second image block, and the third image comprises the third image block; determining a target error value corresponding to each candidate motion vector according to the first image block, the second image block and the third image block, wherein the target error value is obtained based on an error value between the first image block and the second image block and an error value between the first image block and the third image block; and determining a motion vector with the minimum target error value from the plurality of candidate motion vectors as the motion vector corresponding to the first image block according to the target error value corresponding to each candidate motion vector.

20. The apparatus of claim 19, wherein the plurality of candidate motion vectors comprises: one or more preset motion vectors, one or more motion vectors generated randomly, and/or motion vectors corresponding to image blocks adjacent to the first image block.

21. The apparatus of claim 18, wherein the image generation unit is configured to: if the target image has an area which cannot generate an image based on the plurality of image blocks, dividing the third image to obtain the plurality of image blocks of the third image; determining motion vectors respectively corresponding to a plurality of image blocks of the third image according to the first image and the second image; and updating the target image according to the motion vectors corresponding to the image blocks of the third image and the image blocks of the third image to obtain a new target image, wherein the new target image is used for replacing the first image.

22. A terminal device, comprising a processor, a memory, a display screen, a camera, and a bus, wherein:

the processor, the display screen, the camera and the memory are connected through the bus;

the memory is used for storing a computer program and images acquired by the camera;

The processor is configured to control the memory, acquire a computer program stored on the memory and execute the program stored on the memory, and control the display screen to implement the method of any one of claims 1 to 10.

23. A computer readable storage medium comprising a program which, when run on a computer, causes the computer to perform the method of any one of claims 1 to 10.

24. A computer program product comprising instructions which, when run on a terminal, cause the terminal to perform the method of any of claims 1 to 10.