CN116095484B - Video anti-shake method and device and electronic equipment - Google Patents

Abstract

The present application provides a video stabilization method, device and electronic device, which relate to the field of artificial intelligence technology. The video stabilization method can adaptively correct the result obtained by the first path optimization when using EIS for stabilization, thereby alleviating the error accumulation caused by the first path optimization, and then can ensure that a stable image can be output, thereby improving the stabilization effect. In addition, in this method, before performing the first path optimization, the offset of OIS and the position of a preset number of frame images can also be used to predict the position of the next frame image, thereby improving the accuracy of the prediction.

Description

Video anti-shake method, device and electronic equipment

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular to a video stabilization method, device and electronic device.

Background technique

Video shooting is a basic function of electronic devices with cameras (such as mobile phones, tablets, sports cameras, etc.). Shooting videos with high definition and strong image stability is one of the core competitiveness of products. At present, the electronic image stabilization (EIS) technology is mainly used when shooting videos, but in some scenes, such as the scene of image trembling caused by hand shaking and blur under long exposure, the single EIS technology has poor anti-shake effect.

Summary of the invention

The present application provides a video stabilization method, device and electronic device, which can ensure the output of stable images and improve the stabilization effect.

In a first aspect, the present application provides a video stabilization method, the method comprising: obtaining a t-th frame image and a first position corresponding to the t-th frame image, the first position being represented by motion data of a device shooting the t-th frame image when shooting the t-th frame image, and t being a positive integer greater than or equal to 1; inputting the first position and the positions of n frames of images before the t-th frame image into a future path prediction model to obtain a first predicted position corresponding to the (t+1)-th frame image; performing a first path optimization based on the first position, the position of the (n-1) frame image before the t-th frame image, and the first predicted position to obtain a first optimal path, the first optimal path including the ideal positions of each frame image when the (n-1) frame image before the t-th frame image, the t-th frame image, and the (t+1)-th frame image are all stable images. , n is a positive integer greater than or equal to 1; determine that the target deviation value between the position of the target frame image and the predicted position corresponding to the target frame image is greater than the preset deviation threshold, and use the target error correction amount to correct the first ideal position corresponding to the (t+1)th frame image in the first optimal path, the target frame image includes the tth frame image and the n frames before the tth frame image; perform a second path optimization according to the first position and the positions of the n frames before the tth frame image to obtain the second optimal path, the second optimal path includes the ideal positions of each frame image when the (n-1) frames before the tth frame image and the tth frame image are both stable images; process the tth frame image according to the first position and the second ideal position corresponding to the tth frame image in the second optimal path to obtain a stable image. In this way, when using EIS for anti-shake, the result obtained by the first path optimization can be adaptively corrected, thereby alleviating the error accumulation generated by the first path optimization, and reducing the influence of the result of the first path optimization on the second path optimization when processing the next frame image, thereby ensuring that a stable image can be output and improving the anti-shake effect. The result of the first path optimization may be used to determine the boundary for cropping the next frame of image when processing the next frame of image and performing the second path optimization.

In one possible implementation, the target error correction amount is used to correct the first ideal position corresponding to the (t+1)th frame image in the first optimal path, specifically including: using the target error correction amount and a pre-set correction amount scaling ratio to correct the first ideal position corresponding to the (t+1)th frame image in the first optimal path.

In one possible implementation, the target error correction amount is a preset correction amount; or, the target error correction amount is the difference between an ideal position of the (t-1)th frame image in an optimal path obtained by performing a second path optimization on the (t-1)th frame image and an ideal position of the (t-1)th frame image in an optimal path obtained by performing a first path optimization on the (t-1)th frame image.

In a possible implementation, the first position and the positions of the n frames before the t-th frame are input into the future path prediction model to obtain the first predicted position corresponding to the (t+1)-th frame, specifically including: obtaining the offset of the optical image stabilization module on the device that shoots the t-th frame; the first position, the positions and offsets of the n frames before the t-th frame are input into the future path prediction model to obtain the first predicted position corresponding to the (t+1)-th frame. In this way, the "offset of the optical image stabilization module (i.e., OIS)" and the "historical real position of each frame" are used as inputs in the two-level EIS anti-shake, and the motion state of the next frame is predicted before the first path optimization, thereby improving the prediction accuracy.

In a second aspect, the present application provides a video stabilization device, the device comprising: an acquisition module, used to acquire the t-th frame image and a first position corresponding to the t-th frame image, the first position being represented by motion data of a device shooting the t-th frame image when shooting the t-th frame image, and t being a positive integer greater than or equal to 1; a processing module, inputting the first position and the positions of n frames of images before the t-th frame image into a future path prediction model to obtain a first predicted position corresponding to the (t+1)-th frame image; the processing module is also used to perform a first path optimization based on the first position, the position of the (n-1)-th frame image before the t-th frame image, and the first predicted position to obtain a first optimal path, the first optimal path including the ideal positions of each frame image when the (n-1)-th frame image before the t-th frame image, the t-th frame image, and the (t+1)-th frame image are all stable images, and n is A positive integer greater than or equal to 1; a processing module, further used to determine that a target deviation value between the position of a target frame image and a predicted position corresponding to the target frame image is greater than a preset deviation threshold, and using a target error correction amount, correcting a first ideal position corresponding to the (t+1)th frame image in the first optimal path, wherein the target frame image includes the tth frame image and n frames of images before the tth frame image; the processing module is further used to perform a second path optimization according to the first position and the positions of n frames of images before the tth frame image to obtain a second optimal path, wherein the second optimal path includes the (n-1) frames of images before the tth frame image and the ideal positions of each frame image when the tth frame image is a stable image; the processing module is further used to process the tth frame image according to the first position and the second ideal position corresponding to the tth frame image in the second optimal path to obtain a stable image.

In a possible implementation, the processing module is further used to correct the first ideal position corresponding to the (t+1)th frame image in the first optimal path by using the target error correction amount and a preset correction amount scaling ratio.

In one possible implementation, the target error correction amount is a preset correction amount; or, the target error correction amount is the difference between an ideal position of the (t-1)th frame image in an optimal path obtained by performing a second path optimization on the (t-1)th frame image and an ideal position of the (t-1)th frame image in an optimal path obtained by performing a first path optimization on the (t-1)th frame image.

In a possible implementation, the acquisition module is further used to: obtain the offset of the optical image stabilization module on the device that captures the t-th frame image; the processing module is further used to: input the first position, the position and the offset of the n-frame images before the t-th frame image into the future path prediction model to obtain the first predicted position corresponding to the (t+1)-th frame image.

In a third aspect, the present application provides an electronic device, including:

at least one memory for storing a program;

At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed on an electronic device, the electronic device executes the method provided in the first aspect.

In a fifth aspect, the present application provides a computer program product, characterized in that when the computer program product is run on an electronic device, the electronic device executes the method provided in the first aspect.

It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

FIG2 is a schematic diagram of a flow chart of a video anti-shake method provided in an embodiment of the present application;

FIG3 is a schematic diagram of the steps of a video anti-shake method provided by an embodiment of the present application;

FIG4 is a schematic diagram of the steps of another video anti-shake method provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a video stabilization device provided in an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The term "and/or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol "/" in this article indicates that the associated objects are in an or relationship, for example, A/B means A or B.

The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

For example, the electronic image stabilization (EIS) technology mainly analyzes and collects images on the image sensor through a program. When the content of a single-frame image is distorted and stretched, or there are large jumps between consecutive frames of images, that is, the video shakes, the image transformation, moving pixel positions, etc. can be used to eliminate the shake phenomenon, thereby achieving "anti-shake". Its implementation principle is more like "post-processing" of the image.

Generally, the electronic image stabilization technology adopts two-stage electronic image stabilization (Two-Stage EIS). The two-stage electronic image stabilization technology can perform image cropping in two steps while ensuring the anti-shake effect, thereby reducing the amount of calculation and storage of the image processing module before the EIS module in the path. The two-stage electronic image stabilization technology mainly obtains the optimal path for video image stability through two path optimizations, and then adjusts the output video image trajectory to the optimal path, thereby obtaining a video with the best image stability. It can be understood that when the electronic device is moving, the inertial measurement unit (IMU) on it can record the motion state of the electronic device at a certain moment (such as: axis angular velocity, acceleration, etc.). The IMU records the real motion process of the electronic device, that is, the real path of the electronic device. When shooting a video, the motion state of the electronic device at the corresponding moment of each frame can be expressed as a vector, that is, a point on the path. Multiple continuous vectors corresponding to multiple continuous image frames constitute the path of the body movement of the electronic device. The optimal path refers to a continuous, smooth, virtual, and ideal path obtained by processing the real path of the electronic device body using algorithms such as filtering or optimization. In addition, the process of calculating the optimal path is path optimization.

Among them, in the two-level electronic image stabilization technology, the position of the next frame image is generally predicted by the historical data of the body movement of the electronic device. Then, the position of the current frame image, the predicted position of the next frame image, and the position of the preset number of images before the current frame image are used to perform the first path optimization to obtain an optimal path. Then, based on the predicted position of the next frame image and the corresponding position of the next frame image after the first path optimization, the cropping boundary is obtained when the second path optimization is performed on the next frame image. Then, the position of the current frame image and the position of the preset number of images before the current frame image are used to perform the second path optimization to obtain an optimal path. Finally, the image is output based on the actual position of the current frame image and the ideal position of the current frame contained in the optimal path obtained by the second path optimization. This image is the image after the image stabilization process.

Since there is an error in predicting the position of the next frame of image, and the first path optimization needs to use the predicted position of the next frame of image, the optimal path obtained by the first path optimization also has errors. As the errors in the subsequent predictions of the position of the next frame of image accumulate, the optimal path obtained in the subsequent path optimization will become worse and worse, resulting in a worse EIS stabilization effect, that is, a worse anti-shake effect.

In order to solve the problem of poor anti-shake effect when using EIS for anti-shake, the embodiment of the present application provides a video anti-shake method. The video anti-shake method can adaptively correct the result obtained by the first path optimization when using EIS for anti-shake, thereby alleviating the error accumulation caused by the first path optimization, and then can ensure that a stable image can be output, thereby improving the anti-shake effect.

For example, Fig. 1 shows a schematic diagram of the hardware structure of an electronic device. As shown in Fig. 1 , the electronic device 100 includes: a camera module 110, an inertial measurement unit (IMU) module 120, a processor 130 and a memory 140.

Among them, the camera module 110 can be used to capture images. Exemplarily, the camera module 110 may include an image sensor 111. The image sensor 111 can be used to generate an image at a certain moment, and the image can be composed of multiple pixels. In addition, the camera module 110 may also include an optical image stabilization (OIS) module 112. The optical image stabilization module 112 can dynamically adjust the relative position between the image sensor 111 and the lens (not shown in the figure) in the camera module 110 during the process of the image sensor 111 generating an image to achieve mechanical image stabilization, and at the same time output the moving distance of the body of the electronic device 100.

The inertial measurement unit module 120 can be used to record the rotation and translation of the body of the electronic device 100 in real time, and output the body's posture at a certain moment (i.e., the representation of rotation and translation data). The posture over a period of time is the movement path of the body during this period of time. Exemplarily, the inertial measurement unit module 120 may include a gyroscope and/or an accelerometer, etc.

The processor 130 is the computing core and control core of the electronic device 100. The processor 130 may include one or more processing units. For example, the processor 130 may include one or more of an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural network processor (NPU). Among them, different processing units can be independent devices or integrated in one or more processors. Exemplarily, the processor 130 can process the video image through an image processing algorithm, such as processing the brightness, color, pixels, etc. of the video image. In addition, the processor 130 can also calculate various data through an anti-shake algorithm to obtain a stable image.

The memory 140 may store a program, which may be executed by the processor 130 so that the processor 130 executes the method provided in the present application. The memory 140 may also store data. The processor 130 may read the data stored in the memory 140. The memory 140 and the processor 130 may be provided separately. Optionally, the memory 140 may also be integrated in the processor 130.

It is understood that the structure shown in FIG. 1 of the present solution does not constitute a specific limitation on the electronic device 100. In other embodiments of the present solution, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

It can be understood that in the embodiment of the present application, the electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and/or a smart city device, and the embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

Next, based on the above description, the video stabilization method provided in the embodiment of the present application is described in detail.

Exemplarily, FIG2 shows a video stabilization method provided by an embodiment of the present application. The video stabilization method can be executed by the electronic device 100 in FIG1 described above, or by any device, equipment, platform, or device cluster with computing and processing capabilities. For ease of description, the following description is based on the electronic device 100 in FIG1 as the execution subject. Specifically, as shown in FIG2, the method includes the following steps:

S201. Acquire a t-th frame image and a first position corresponding to the t-th frame image, where the first position is represented by motion data of the electronic device 100 when shooting the t-th frame image, and t is a positive integer greater than or equal to 1.

Specifically, the electronic device 100 can obtain the tth frame image through the image sensor 111 thereon, and can obtain the motion data (such as acceleration data, angular velocity data, etc.) of the electronic device 100 when taking the tth frame image through the IMU module 120 thereon, and then obtain the first position corresponding to the tth frame image.

S202 , input the first position and the positions of n frames of images before the t-th frame of image into a future path prediction model to obtain a first predicted position of the path of the (t+1)-th frame of image.

Specifically, after the first position is acquired, the first position and the positions of n frames before the tth frame may be input into a future path prediction model to obtain a first predicted position of the path of the (t+1)th frame.

As a possible implementation method, since the offset detected by the optical image stabilization module 112 on the electronic device 100 can also reflect the movement state of the electronic device 100, the offset detected by the optical image stabilization module 112 on the electronic device 100 can also be input into the future path prediction model to obtain the first predicted position, thereby improving the accuracy of the path prediction.

S203. Perform a first path optimization based on the first position, the position of the (n-1) frame image before the t-th frame image and the first predicted position to obtain a first optimal path, wherein the first optimal path includes the ideal positions of each frame image when the (n-1) frame image before the t-th frame image, the t-th frame image and the (t+1)-th frame image are all stable images, and n is a positive integer greater than or equal to 1.

Specifically, after predicting the first predicted path of the path of the (t+1)th frame image, the first path optimization can be performed based on the first position, the position of the n frames before the tth frame image, and the first predicted position to obtain the first optimal path. Exemplarily, the first position, the position of the n frames before the tth frame image, and the first predicted position can be input into the path optimization model to perform the first path optimization and obtain the first optimal path. In addition, a filtering algorithm (such as mean filtering, etc.) can also be used to process the first position, the position of the n frames before the tth frame image, and the first predicted position to obtain the first optimal path.

S204, determining whether a target deviation value between the position of a target frame image and a predicted position corresponding to the target frame image is greater than a preset deviation threshold, the target frame image comprising the t-th frame image and n frames of images before the t-th frame image.

Specifically, after the first path optimization is performed, it can be determined whether the target deviation value between the position of the target frame image and the predicted position corresponding to the target frame image is greater than a preset deviation threshold. The target frame image includes the t-th frame image and the n-th frame image before the t-th frame image. Exemplarily, the deviation value between the position of each frame image and its corresponding predicted position can be determined first, and then the average value, variance value, etc. of all the deviation values are used as the target deviation value.

When the target deviation value is greater than the preset deviation threshold, it indicates that the deviation between the ideal position corresponding to the (t+1)th frame image in the first optimal path obtained by the first path optimization and its actual path is large, and at this time, the ideal position corresponding to the (t+1)th frame image needs to be corrected, that is, S205 is executed. Otherwise, S206 is executed.

S205 , using the target error correction amount, correcting the first ideal position corresponding to the (t+1)th frame image in the first optimal path.

Specifically, a preset error correction amount may be used to correct the first ideal position corresponding to the (t+1)th frame image in the first optimal path.

In addition, the correction may be performed using the difference between the ideal position of the (t-1)th frame image in the optimal path obtained by performing the second path optimization on the (t-1)th frame image and the ideal position of the (t-1)th frame image in the optimal path obtained by performing the first path optimization. That is, the target error correction amount may be a preset value or the difference between the ideal position of the (t-1)th frame image in the optimal path obtained by performing the second path optimization on the (t-1)th frame image and the ideal position of the (t-1)th frame image in the optimal path obtained by performing the first path optimization.

In one example, the shrinkage boundary corresponding to the (t+1)th frame image can also be determined based on the first ideal position and the first predicted position. Specifically, the shrinkage boundary corresponding to the (t+1)th frame image can be determined based on the first ideal position and the first predicted position. Exemplarily, the shrinkage boundary can be data obtained by subtracting the absolute value between the first ideal position and the first predicted position from a preset boundary threshold. In addition, the shrinkage boundary corresponding to each frame image can be used to crop the frame image when processing the frame image (such as when performing a second path optimization) to reduce the image storage and calculation amount and improve processing efficiency. Exemplarily, the preset boundary threshold can be the boundary for cropping the frame image when performing the first path optimization on the frame image to be processed.

S206. Perform a second path optimization based on the first position and the positions of n frames before the t-th frame to obtain a second optimal path, wherein the second optimal path includes the ideal positions of each frame when the (n-1) frames before the t-th frame and the t-th frame are stable images.

Specifically, a second path optimization can be performed based on the first position and the position of the n-frame images before the t-th frame image to obtain a second optimal path. The second optimal path includes the ideal positions of each frame image when the (n-1)-frame image before the t-th frame image and the t-th frame image are both stable images. Exemplarily, the first position and the position of the n-frame image before the t-th frame image can be input into the path optimization model, the first path optimization is performed, and the second optimal path is obtained. In addition, a filtering algorithm (such as mean filtering, etc.) can also be used to process the first position and the position of the n-frame image before the t-th frame image to obtain the second optimal path.

S207 . Process the t-th frame image according to the first position and the second ideal position corresponding to the t-th frame image in the second optimal path to obtain a stable image.

Specifically, after obtaining the second optimal path, the t-th frame image can be processed according to the second ideal position corresponding to the t-th frame image in the first position and the second optimal path to obtain a stable image. Exemplarily, the t-th frame image and the second ideal position can be input into the Rodriguez rotation matrix formula to obtain the relationship between the pixel coordinates on the t-th frame image before and after the change, that is, the matrix of the coordinate transformation, and then the pixels of the t-th frame image can be moved, and a stable image can be output, thereby achieving the purpose of EIS anti-shake. It can be understood that since EIS adopts a software algorithm that obtains the pixel movement transformation of the picture based on the motion state transformation from the position to the optimal path, and the formula for calculating the pixel movement transformation from the motion state transformation is the Rodriguez rotation matrix formula, a stable image can be obtained by the Rodriguez rotation matrix formula.

Therefore, in the video stabilization method provided in the embodiment of the present application, in the process of using the EIS stabilization technology, an adaptive error correction technology is adopted to dynamically adjust the first path optimization result, thereby eliminating the error accumulation problem during the first path optimization and improving the two-level stabilization effect.

The above is a description of the video stabilization method provided in the embodiment of the present application. For ease of understanding, the following examples are given for illustration.

For example, Fig. 3 shows a schematic diagram of a video anti-shake method. As shown in Fig. 3, the method specifically includes the following steps:

S301: Initialization.

Specifically, the initialization may include: error correction amount Δs, correction start threshold e, correction amount scaling parameter α, camera intrinsic parameter matrix K, optimized boundary threshold σ, Rodriguez formula R, etc.

Among them, the correction amount scaling parameter α can be understood as the strength of correction; when the correction amount scaling parameter α is large, the correction strength is large, and when the correction amount scaling parameter α is small, the correction strength is small.

The camera intrinsic parameter matrix K is a parameter determined by the physical characteristics of the camera itself on the electronic device, and can be obtained by a classic camera calibration algorithm (such as Zhang Zhengyou calibration method).

The optimized boundary threshold σ is the cropping ratio that needs to be cropped when using the EIS algorithm. Since the essence of the EIS algorithm is to move the pixels of the input image, after moving the pixels, the entire image is not a regular rectangle, but needs to output a rectangle of a specified resolution, so the boundary needs to be cropped, and this cropping ratio is the boundary threshold parameter.

In one example, S301 does not have to be executed every time, and it can be selected according to actual conditions, which is not limited here.

S302: Obtain the tth frame image and the corresponding position/>

S303: Predict the position of the next frame to obtain the position of the next frame

Specifically, the t-th frame image and the corresponding position/> And the positions of n images before the t-th frame image are used as input to the pre-trained future path prediction model to obtain the predicted position of the next frame/> For example, taking n=29 as an example, it is to set/> Input into the pre-trained future path prediction model to get the predicted position of the next frame/>

S304: First path optimization.

Specifically, the position corresponding to the t-th frame image The position of the (n-1)th image before the t-th frame, the position of the next frame/> As input, the first path optimization is performed to obtain the optimal path of the first optimization, which includes the ideal position of the t-th frame image and the ideal position of the next frame image. For example, taking n=29 as an example, the position used as input is / > When performing the first path optimization, the boundary threshold of each frame is σ, and the optimal path output for the first optimization is/>

In one example, when performing the first path optimization, the optimized boundary threshold σ at the time of initialization can be used to perform the t-th frame image Clipping can reduce the amount of computation in subsequent processing and improve processing efficiency.

S305: Determine whether correction is turned on.

Specifically, the difference between the position of the (n+1) frame image including the tth frame and the corresponding predicted position can be calculated, and then the difference is used to determine whether to start the correction. For example, taking n=29 as an example, the predicted average error If the error is greater than the opening threshold e, step S306 is executed, otherwise step S306 is skipped.

S306: Correct the ideal position of the next frame in the first optimal path.

Specifically, the initialization error correction amount Δs and the correction amount scaling parameter α can be used to correct the ideal position of the next frame in the first optimal path obtained. For example, the ideal position of the next frame in the first optimal path after correction is

S307: Obtain the shrinkage boundary of the next frame of image.

Specifically, since the essence of the EIS algorithm is to move the pixels of the input image, after moving the pixels, the entire image is not a regular rectangle, but needs to output a rectangle of a specified resolution, so the border needs to be cropped. In addition, in the two-level anti-shake, the first path optimization is performed, and part of the croppable border is used. Then, when performing the second path optimization, the croppable border range is reduced, so the border needs to be shrunk.

Among them, the position of the t-th frame predicted when processing the (t-1) frame image can be and the first corrected optimal path when processing the (t-1) frame image/> Shrink the boundary threshold of frame t. New boundary threshold In addition, the boundary threshold of the (t+1)th frame image can also be shrunk at this time to facilitate the subsequent processing of the (t+1)th frame image. The boundary threshold of the (t+1)th frame image is Wherein, σ _t+1 may be the optimization boundary threshold σ during initialization.

S308: Second path optimization.

Specifically, the position corresponding to the t-th frame image The positions of n images before the t-th frame image are used as input, and the second path optimization is performed to obtain the optimal path of the second optimization, which includes the ideal position of the t-th frame image. For example, taking n=29 as an example, the position used as input is / > When performing the second path optimization, the boundary thresholds of each frame are (∈ _t-29 ,…,∈ _t-1 ,∈ _t ), and the optimal path of the second optimization is outputted

In one example, when performing the second path optimization, the contraction boundary ∈ _t of the t-th frame image obtained when processing the (t-1)-th frame image can be used to crop the t-th frame image after the first path optimization. Perform cropping again to reduce the amount of computation required for subsequent processing and improve processing efficiency.

S309: Calculate the error correction amount.

Specifically, the position difference corresponding to the t-th frame image in the optimal path of the two path optimizations can be used as the error correction amount: The error correction amount calculated here is the error correction amount when processing the next frame of image.

S310: Outputting an anti-shake image.

Specifically, the position in the optimal anti-shake path corresponding to the current t-th frame image is obtained. After that, the optimal anti-shake path for the t-th frame image can be obtained by using the Rodriguez rotation matrix formula R and the camera internal parameter matrix. and its real location/> Processing, get the transformation matrix/> Then, the obtained transformation matrix H is applied to the t-th frame image/> You can get a stable image by

Finally, after outputting the anti-shake image, the process may jump to S302 to loop through the next frame. Specifically, after processing the tth frame image, the process may start to process the (t+1)th frame image. When jumping to S302, t may be updated to (t+1).

For example, another process diagram of a video anti-shake method is shown in Fig. 4. The main difference between Fig. 4 and Fig. 3 is that the offset detected by the OIS module on the electronic device is added when predicting the path of a future frame of image in Fig. 4.

As shown in FIG4 , the specific steps include:

S401: Initialization.

Specifically, the initialization may include: initializing the error correction amount Δs, the correction start threshold e, the correction amount scaling parameter α, the camera intrinsic parameter matrix K, the optimization boundary threshold σ, the Rodriguez formula R, etc.

In one example, S401 does not have to be executed every time, and it can be selected according to actual conditions, which is not limited here.

S402: Obtaining the tth frame image and the corresponding position/>

S403: OIS offset guides the path prediction for the next frame to obtain the position of the next frame

Specifically, the t-th frame image and the corresponding position/> The positions of n images before the t-th frame image and the offset f detected by the OIS module on the electronic device are used as input to the pre-trained future path prediction model to obtain the predicted position of the next frame/> For example, taking n=29 as an example, it is to set/> Input into the pre-trained future path prediction model to get the predicted position of the next frame/>

S404: First path optimization.

Specifically, the position corresponding to the t-th frame image The position of the (n-1)th image before the t-th frame, the position of the next frame/> As input, the first path optimization is performed to obtain the optimal path of the first optimization, which includes the ideal position of the t-th frame image and the ideal position of the next frame image. For example, taking n=29 as an example, the position used as input is / > When performing the first path optimization, the boundary threshold of each frame is σ, and the optimal path output for the first optimization is/>

In one example, when performing the first path optimization, the optimized boundary threshold σ at the time of initialization can be used to perform the t-th frame image Clipping can reduce the amount of computation in subsequent processing and improve processing efficiency.

S405: Determine whether correction is enabled.

Specifically, the difference between the position of the (n+1) frame image including the tth frame and the corresponding predicted position can be calculated, and then the difference is used to determine whether to start the correction. For example, taking n=29 as an example, the predicted average error If the error is greater than the opening threshold e, step S406 is executed, otherwise step S406 is skipped.

S406, correcting the first optimal path

Specifically, the initialization error correction amount Δs and the correction amount scaling parameter α can be used to correct the ideal position of the next frame in the first optimal path obtained. For example, the ideal position of the next frame in the first optimal path after correction is

S407: Obtain the shrinkage boundary of the next frame of image.

Specifically, since the essence of the EIS algorithm is to move the pixels of the input image, after moving the pixels, the entire image is not a regular rectangle, and we need to output a rectangle with a specified resolution, so we need to crop the border. In addition, in the two-level anti-shake, the first path optimization is performed, and part of the croppable border is used. Then, when performing the second path optimization, the croppable border range is reduced, so the border needs to be shrunk.

Among them, the position of the t-th frame predicted when processing the (t-1) frame image can be and the first corrected optimal path when processing the (t-1) frame image/> Shrink the boundary threshold of frame t. New boundary threshold In addition, the boundary threshold of the (t+1)th frame image can also be shrunk at this time to facilitate the subsequent processing of the (t+1)th frame image. The boundary threshold of the (t+1)th frame image is Wherein, σ _t+1 may be the optimization boundary threshold σ during initialization.

S408: Second path optimization.

Specifically, the position corresponding to the t-th frame image The positions of n images before the t-th frame image are used as input, and the second path optimization is performed to obtain the optimal path of the second optimization, which includes the ideal position of the t-th frame image. For example, taking n=29 as an example, the position used as input is / > When performing the second path optimization, the boundary thresholds of each frame are (∈ _t-29 ,…,∈ _t-1 ,∈ _t ), and the optimal path of the second optimization is outputted Among them, each value in the optimal path is the ideal position corresponding to each frame image.

In one example, when performing the second path optimization, the contraction boundary ∈ _t of the t-th frame image obtained when processing the (t-1)-th frame image can be used to crop the t-th frame image after the first path optimization. Perform cropping again to reduce the amount of computation required for subsequent processing and improve processing efficiency.

S409: Calculate the error correction amount.

Specifically, the position difference corresponding to the t-th frame image in the optimal path of the two path optimizations can be used as the error correction amount: The error correction amount calculated here is the error correction amount when processing the next frame of image.

S410: Outputting an anti-shake image.

Specifically, the position in the optimal anti-shake path corresponding to the current t-th frame image is obtained. After that, the optimal anti-shake path for the t-th frame image can be obtained by using the Rodriguez rotation matrix formula R and the camera internal parameter matrix. and its real location/> Processing, get the transformation matrix/> Then, the obtained transformation matrix H is applied to the t-th frame image/> You can get a stable image by

Finally, after outputting the anti-shake image, the process may jump to S402 to loop through the next frame. Specifically, after processing the tth frame image, the process may start to process the (t+1)th frame image. When jumping to S402, t may be updated to (t+1).

It is understandable that the method provided in any embodiment of the present application, in addition to being executed by the electronic device 100, can also be executed by any device, equipment, platform, and device cluster with computing and processing capabilities, wherein the execution method of the execution subject other than the electronic device 100 is also within the protection scope of the present application. In addition, the execution order of each step in any embodiment of the present application can be adjusted according to the actual situation without contradiction, and the adjusted technical solution is also within the scope of the present application. In addition, the various steps in any embodiment of the present application can also be selectively executed, which is not limited here. In addition, all or part of any features of any embodiment of the present application can be freely and arbitrarily combined without contradiction, and the combined technical solution is also within the protection scope of the present application.

Based on the method in the above embodiment, the embodiment of the present application further provides a video anti-shake device. Please refer to FIG. 5, which is a schematic diagram of the structure of a video anti-shake device provided in the embodiment of the present application. As shown in FIG. 5, the video anti-shake device 500 includes one or more processors 501 and an interface circuit 502. Optionally, the video anti-shake device 500 may also include a bus 503.

in:

Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in processor 501 or an instruction in software form. The above-mentioned processor 501 can be a general-purpose processor, a digital communicator (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods and steps in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The interface circuit 502 can be used for sending or receiving data, instructions or information, and the processor 501 can use the data, instructions or other information received by the interface circuit 502 to process, and the processing completion information can be sent out through the interface circuit 502.

Optionally, the video stabilization device further includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM). Optionally, the memory stores an executable software module or a data structure, and the processor may perform a corresponding operation by calling an operation instruction stored in the memory (the operation instruction may be stored in the operating system).

Optionally, the interface circuit 502 may be used to output the execution result of the processor 501 .

It should be noted that the functions corresponding to the processor 501 and the interface circuit 502 can be implemented through hardware design, software design, or a combination of hardware and software, which is not limited here.

It should be understood that each step of the above method embodiment can be completed by a hardware logic circuit or a software instruction in a processor. Among them, the video anti-shake device can be applied to the electronic device 100 in Figure 2 above to implement the method provided in the embodiment of the present application.

It is understandable that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims (10)

1. A method for video anti-shake, the method comprising:

acquiring a t frame image and a first position corresponding to the t frame image, wherein the first position is represented by motion data of equipment for shooting the t frame image when shooting the t frame image, and t is a positive integer greater than or equal to 1;

inputting the first position and the position of the n frame images before the t frame image into a future path prediction model to obtain a first prediction position corresponding to the (t+1) th frame image;

performing a first path optimization according to the first position, the position of the (n-1) frame image before the t frame image and the first prediction position to obtain a first optimal path, wherein the first optimal path comprises ideal positions of all frame images when the (n-1) frame image before the t frame image, the t frame image and the (t+1) frame image are stable images, and n is a positive integer greater than or equal to 1;

determining that a target deviation value between a position of a target frame image and a predicted position corresponding to the target frame image is larger than a preset deviation threshold value, and correcting a first ideal position corresponding to the (t+1) th frame image in the first optimal path by using a target error correction amount, wherein the target frame image comprises the t frame image and n frame images before the t frame image;

Performing a second path optimization according to the first position and the position of the n frame image before the t frame image to obtain a second optimal path, wherein the second optimal path comprises ideal positions of all frame images when the (n-1) frame image before the t frame image and the t frame image are stable images;

and processing the t frame image according to the first position and a second ideal position corresponding to the t frame image in the second optimal path to obtain a stable image.

2. The method according to claim 1, wherein the correcting the first ideal position corresponding to the (t+1) th frame image in the first optimal path by using the target error correction amount specifically includes:

and correcting a first ideal position corresponding to the (t+1) th frame image in the first optimal path by utilizing the target error correction amount and a preset correction amount scaling ratio.

3. The method according to claim 1 or 2, wherein the target error correction amount is a preset correction amount;

or the target error correction amount is the difference between the ideal position of the (t-1) th frame image in the optimal path obtained by carrying out the second path optimization on the (t-1) th frame image and the ideal position of the (t-1) th frame image in the optimal path obtained by carrying out the first path optimization on the (t-1) th frame image.

4. The method according to claim 1 or 2, wherein the step of inputting the first position and the position of the n-frame image before the t-frame image into a future path prediction model to obtain a first predicted position corresponding to the (t+1) -th frame image specifically includes:

acquiring the offset of an optical anti-shake module on equipment for shooting the t-th frame image;

and inputting the first position, the position of the n frame images before the t frame image and the offset into a future path prediction model to obtain a first prediction position corresponding to the (t+1t+1) th frame image.

5. A video anti-shake apparatus, the apparatus comprising:

the acquisition module is used for acquiring a t frame image and a first position corresponding to the t frame image, wherein the first position is represented by motion data of equipment for shooting the t frame image when shooting the t frame image, and t is a positive integer greater than or equal to 1;

the processing module inputs the first position and the position of the n frame images before the t frame image into a future path prediction model to obtain a first prediction position corresponding to the (t+1t+1) th frame image;

the processing module is further configured to perform a first path optimization according to the first position, the position of the (n-1) frame image before the t frame image, and the first prediction position, so as to obtain a first optimal path, where the first optimal path includes ideal positions of each frame image when the (n-1) frame image before the t frame image, and the (t+1t+1) th frame image are all stable images, and n is a positive integer greater than or equal to 1;

The processing module is further configured to determine that a target deviation value between a position of a target frame image and a predicted position corresponding to the target frame image is greater than a preset deviation threshold, and correct a first ideal position corresponding to the (t+1t+1) th frame image in the first optimal path by using a target error correction amount, where the target frame image includes the t-th frame image and an n-frame image before the t-th frame image;

the processing module is further configured to perform a second path optimization according to the first position and the position of the n frame image before the t frame image, to obtain a second optimal path, where the second optimal path includes ideal positions of each frame image when the (n-1) frame image before the t frame image and the t frame image are both stable images;

the processing module is further configured to process the t frame image according to the first position and a second ideal position corresponding to the t frame image in the second optimal path, so as to obtain a stable image.

6. The apparatus of claim 5, wherein the processing module is further configured to:

and correcting a first ideal position corresponding to the (t+1) th frame image in the first optimal path by utilizing the target error correction amount and a preset correction amount scaling ratio.

7. The apparatus according to claim 5 or 6, wherein the target error correction amount is a preset correction amount;

or the target error correction amount is the difference between the ideal position of the (t-1) th frame image in the optimal path obtained by carrying out the second path optimization on the (t-1) th frame image and the ideal position of the (t-1) th frame image in the optimal path obtained by carrying out the first path optimization on the (t-1) th frame image.

8. The apparatus of claim 5 or 6, wherein the device comprises a plurality of sensors,

the acquisition module is further configured to: acquiring the offset of an optical anti-shake module on equipment for shooting the t-th frame image;

the processing module is further configured to: and inputting the first position, the position of the n frame images before the t frame image and the offset into a future path prediction model to obtain a first prediction position corresponding to the (t+1) th frame image.

9. An electronic device, comprising:

at least one memory for storing a program;

at least one processor for executing a memory-stored program, which processor is adapted to perform the method according to any of claims 1-4 when the memory-stored program is executed.

10. A computer readable storage medium storing a computer program which, when run on an electronic device, causes the electronic device to perform the method of any one of claims 1-4.