CN113038271A - Video automatic editing method, device and computer storage medium - Google Patents

Abstract

The embodiments of the present application disclose an automatic video editing method, device, and computer storage medium, so that the video generated by the editing can maximize the information of the target person and avoid the information of other irrelevant people. The embodiment of the present application includes: applying the calculated quantized value of attitude information and the change value of optical flow energy to the calculation of the reward value of the action in the reinforcement learning algorithm, and determining the candidate video frame corresponding to the maximum reward value as the current video frame. The next video frame, taking the next video frame of the current video frame as the new current video frame, returns to the step of performing the calculation of the reward value of the action under the current video frame, at the same time, based on the position and size of the target person in the video frame The size of the video picture window is determined, and the video picture about the target person is extracted according to the video picture window, so that the final synthesized video can maximize the information about the target person and avoid showing other irrelevant person information.

Description

Video automatic editing method, device and computer storage medium

technical field

The embodiments of the present application relate to the field of video editing, and in particular, to an automatic video editing method, device, and computer storage medium.

Background technique

In the prior art, automatic video editing can improve the work efficiency of video editing in the fields of security, education, film and television entertainment, and the like. After the video is edited, the data volume of the video is greatly reduced, and the storage space occupied by the video is reduced. Therefore, automatic video editing can also alleviate the storage problem of massive videos. After the video is automatically edited, more storage space can be released.

The existing automatic video editing system is mainly designed for videos such as dance party videos, concert videos, outdoor activity videos, and football game videos, focusing on making the video content richer and more diverse to increase interest and performance. Improve perception. However, in some scenes where the target person needs to be highlighted in the video, the existing automatic video editing system cannot handle it well, because the existing automatic video editing system focuses on presenting more video content and cannot focus On the target person, no more information about the target person can be presented. At the same time, the existing video automatic editing system presents the information of other characters unrelated to the target character in the edited video, which may also cause the privacy leakage of other characters in the video.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an automatic video editing method, device, and computer storage medium, so that the video generated by the editing can maximize the presentation of the information of the target person and avoid the presentation of the information of other irrelevant persons.

A first aspect of the embodiments of the present application provides an automatic video editing method, and the method includes:

Calculate the facial posture information of the target person of each video frame in at least one video, calculate the posture information quantization value corresponding to the facial posture information, and calculate the optical flow energy change value of each video frame;

Use any video frame in any video as the current video frame;

Based on the reinforcement learning algorithm, the reward value of the action under the current video frame is calculated according to the pose information quantization value and the optical flow energy change value of the current video frame, and the candidate video frame corresponding to the maximum reward value is determined as the current video frame. The next video frame, taking the next video frame of the current video frame as the new current video frame, and returning to execute the reinforcement learning algorithm, according to the pose information quantization value of the current video frame and the optical flow energy change value calculated at the current video frame. The step of the reward value of the action under the current video frame; wherein, the action is to select a candidate video frame from each video of the at least one video respectively;

Determine a video frame sequence according to the sequence of the current video frames, and obtain an initial composite video based on the video frame sequence;

According to the position and size of each frame of the target character in the initial composite video, determine the position and size of the video picture window of each frame in the initial composite video;

Based on the position and size of the video picture window, a video picture of each frame in the initial composite video is extracted to obtain a target composite video.

A second aspect of the embodiments of the present application provides an automatic video editing device, and the device includes:

A calculation unit, used to calculate the facial posture information of the target person in each video frame in at least one video, calculate the quantized value of posture information corresponding to the facial posture information, and calculate the optical flow energy change of each video frame. value;

Determining unit, for taking any video frame in any video as the current video frame;

The editing unit is used to calculate the reward value of the action under the current video frame according to the pose information quantization value and the optical flow energy change value of the current video frame based on the reinforcement learning algorithm, and determine the candidate video frame corresponding to the maximum reward value as the selected video frame. Describe the next video frame of the current video frame, take the next video frame of the current video frame as the new current video frame, and return to execute the described based reinforcement learning algorithm, according to the attitude information quantization value and optical flow energy of the current video frame The step of calculating the reward value of the action under the current video frame by the change value; wherein, the action is to select a candidate video frame from each video of the at least one video respectively;

a generating unit, configured to determine a sequence of video frames according to the sequence of the current video frames, and obtain an initial composite video based on the sequence of video frames;

The determining unit is further configured to determine the position and size of the video picture window of each frame in the initial composite video according to the position and size of each frame of the target character in the initial composite video;

An extraction unit, configured to extract the video picture of each frame in the initial composite video based on the position and size of the video picture window to obtain a target composite video.

A third aspect of the embodiments of the present application provides an automatic video editing device, and the device includes:

The memory is used to store the computer program; the processor is used to implement the steps of the automatic video editing method according to the first aspect when the computer program is executed.

A fourth aspect of the embodiments of the present application provides a computer storage medium, where an instruction is stored in the computer storage medium, and when the instruction is executed on a computer, the computer executes the method of the foregoing first aspect.

As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

In this embodiment, by calculating the quantized value of the facial posture information and the optical flow energy change value of the target person in each video frame, the calculated quantized value of the posture information and the optical flow energy change value are applied to the reinforcement learning algorithm for action The calculation of the reward value of , and determine the candidate video frame corresponding to the maximum reward value as the next video frame of the current video frame, take the next video frame of the current video frame as the new current video frame, and return to perform the calculation in the current video frame. The step of the reward value of the lower action, so that each video frame determined from at least one video can maximize the information of the target person and avoid showing the screen where the target person is occluded. At the same time, a video picture window is determined based on the position and size of the target person in the video frame, and a video picture about the target person is extracted according to the video picture window, so that the final synthesized video can maximize the information about the target person, and Avoid presenting information about other irrelevant characters.

Description of drawings

1 is a schematic flowchart of a method for automatic video editing in the embodiment of the application;

2 is another schematic flowchart of the automatic video editing method in the embodiment of the application;

3 is a schematic diagram of a facial gesture information in an embodiment of the present application;

4 is a schematic structural diagram of an automatic video editing device in an embodiment of the application;

FIG. 5 is another schematic structural diagram of an apparatus for automatic video editing according to an embodiment of the present application.

Detailed ways

Embodiments of the present application provide an automatic video editing method, device, and computer storage medium, so that the video generated by the editing can maximize the presentation of the information of the target person and avoid the presentation of the information of other irrelevant persons.

Referring to FIG. 1, an embodiment of the automatic video editing method in the embodiment of the present application includes:

101. Calculate the facial posture information of the target character of each video frame in at least one video, calculate the posture information quantization value corresponding to the facial posture information, and calculate the optical flow energy change value of each video frame;

The method of this embodiment can be applied to an automatic video editing apparatus, and the automatic video editing apparatus may specifically be a computer device with data processing capability, such as a terminal and a server.

In the application scenario where the target person in the video needs to be emphasized, this embodiment will acquire at least one video, wherein the video picture of each video includes the target person, and the task of this embodiment is to automatically edit the at least one video, In order to make the video generated by the clip focus on the information of the target person and objects interacting with the target person, and ensure that the information of other irrelevant people is not displayed in the video generated by the clip, so as to protect the privacy information of other irrelevant people.

After acquiring at least one video, the facial posture information of the target person in each video frame of each video is calculated, and the quantized value of posture information corresponding to the facial posture information is calculated. In addition, this embodiment also proposes a method of determining the occlusion of the target person in the video picture, that is, determining the occlusion situation of the target person in the video picture according to the change value of the optical flow energy. Therefore, in this step, the optical flow energy change value of each video frame is also calculated, and the occlusion situation of the target person is reflected by the optical flow energy change value.

102. Use any video frame in any video as the current video frame;

In this embodiment, a reinforcement learning algorithm is used to determine each frame in the video generated by the clip, and the video frame is used as a state in the reinforcement learning algorithm. The user can designate any video frame in any video as the current video frame, so the automatic video editing device determines the current video frame according to the user's designation, then the current video frame is regarded as a state in the reinforcement learning algorithm, and in the subsequent steps, the The next state is determined according to the state corresponding to the current video frame.

103. Based on the reinforcement learning algorithm, calculate the reward value of the action under the current video frame according to the pose information quantization value of the current video frame and the optical flow energy change value, and determine the candidate video frame corresponding to the maximum reward value as the next video frame of the current video frame. Video frame, take the next video frame of the current video frame as the new current video frame, and return to execute the algorithm based on reinforcement learning, according to the pose information quantization value of the current video frame and the optical flow energy change value to calculate the action under the current video frame. return value steps;

In the automatic editing process, this embodiment sequentially determines each video frame in the video generated by the editing. Specifically, after determining the current video frame, the automatic video editing device calculates the reward value of the action under the current video frame according to the quantized value of the pose information and the optical flow energy change value of the current video frame based on the reinforcement learning algorithm.

The reinforcement learning algorithm in this embodiment may specifically be a Markov decision process. In the reinforcement learning algorithm, the larger the reward value of the action, the more meaningful the action is. The virtual agent in the reinforcement learning algorithm will optimize the strategy according to the action corresponding to the maximum reward value, and then take the next step according to the optimized strategy. Actions. Therefore, after calculating the reward values of multiple actions under the current video frame, the candidate video frame selected by the action with the largest reward value is determined as the next video frame of the current video frame.

After determining the next video frame of the current video frame, take the next video frame of the current video frame as the new current video frame, and return to execute the reinforcement learning algorithm based on the quantization value of the pose information of the current video frame and the change of the optical flow energy Value Steps to compute the reward value for the action at the current video frame. The action under the current video frame refers to selecting a candidate video frame from each video of at least one video respectively.

104. Determine a video frame sequence according to the sequence of the current video frames, and obtain an initial composite video based on the video frame sequence;

Through step 103, each video frame can be sequentially determined, and the determined multiple video frames have a sequential determination order. Therefore, the video frame sequence can be determined according to the sequential determination order of the current video frame in step 103, and then based on the video frame sequence. Get the initial composite video.

105, according to the position and size of each frame of the target character in the initial composite video, determine the position and size of the video picture window of each frame in the initial composite video;

After the initial composite video is obtained, since the purpose of this embodiment is to focus on the target person in the video picture, the position and size of the target person in each frame of the initial composite video are further determined, and according to the target person in each frame The position and size of a frame of picture determine the position and size of the video picture window of each frame in the initial composite video.

106. Extract the video picture of each frame in the initial composite video based on the position and size of the video picture window to obtain the target composite video;

After determining the position and size of the video picture window, extract the video picture of each frame in the initial composite video based on the position and size of the video picture window, so as to obtain the target composite video and realize automatic video editing.

In this embodiment, by calculating the quantized value of the facial posture information and the optical flow energy change value of the target person in each video frame, the calculated quantized value of the posture information and the optical flow energy change value are applied to the reinforcement learning algorithm for action The calculation of the reward value of , and determine the candidate video frame corresponding to the maximum reward value as the next video frame of the current video frame, take the next video frame of the current video frame as the new current video frame, and return to perform the calculation in the current video frame. The step of the reward value of the lower action, so that each video frame determined from at least one video can maximize the information of the target person and avoid showing the screen where the target person is occluded. At the same time, a video picture window is determined based on the position and size of the target person in the video frame, and a video picture about the target person is extracted according to the video picture window, so that the final synthesized video can maximize the information about the target person, and Avoid presenting information about other irrelevant characters.

Based on the foregoing embodiment shown in FIG. 1 , the embodiments of the present application will be described in further detail below. Referring to FIG. 2, another embodiment of the automatic video editing method in the embodiment of the present application includes:

201. Calculate the facial posture information of the target character of each video frame in at least one video, calculate the posture information quantization value corresponding to the facial posture information, and calculate the optical flow energy change value of each video frame;

In this embodiment, the facial posture information of the target person in each video frame can be calculated according to the facial posture estimation algorithm. Specifically, the face pose information calculated by the face pose estimation algorithm can be represented by a rotation matrix, a rotation vector, a quaternion or an Euler angle. Since Euler angles are more readable, Euler angles can preferably be used to represent facial pose information. As shown in FIG. 3 , the facial posture information of the target person, such as the pitch angle, the yaw angle and the roll angle, can be calculated according to the facial posture estimation algorithm.

In order to keep the calculated facial posture information consistent with the symmetrical structure of the human face, the present embodiment adopts a multivariate Gaussian model to calculate the quantized value of posture information corresponding to the facial posture information. Further, for the convenience of calculation, the calculated quantized value of attitude information can be normalized, and the normalized quantized value of attitude information can be used in the subsequent calculation process.

Specifically, the specific method of calculating the optical flow energy change value in this embodiment is to calculate the optical flow information of each video frame in at least one video and calculate the optical flow information of other video frames belonging to the same video as each video frame. , calculate the optical flow energy of each video frame according to the optical flow information of each video frame, calculate the optical flow energy of the other video frames according to the optical flow information of the other video frames, calculate the relationship between each video frame and the other video frames The optical flow energy difference value of the video frame and the interval time between each video frame and the other video frames, the quotient of the optical flow energy difference value and the interval time is taken as the optical flow energy change value of each video frame.

For example, assuming that the automatic video editing device obtains C-channel videos (C≥1), each channel of video includes T frames of video frames, and a certain video frame in the C-channel video can be expressed as f _{c, t} (c=1,..., C; t=1, . . . , T), then the video frame that belongs to the same channel of video as f _{c, t and is adjacent to f c, t+1 can} be expressed as f _{c, t+1} . Calculate the optical flow information of f c, _t and the optical flow information of f _{c, t+1} respectively _, calculate the optical flow energy of f c, t according to the optical flow information of f _{c, t} , according to the light of f _{c, t+1} The flow information calculates the optical flow energy of f _{c, t+1} , calculates the optical flow energy difference between the optical flow energy of f _{c, t} and the optical flow energy of f _{c, t+1} , as well as f _{c, t} and f _{c, t +1} interval time, and then take the quotient of the calculated optical flow energy difference and the interval time as the optical flow energy change value of f _{c, t} .

In the multivariate Gaussian model, when the angle of the Euler angle of the face tends to 0, the quantized value of the posture information corresponding to the facial posture information is the largest; when the face is deflected and the Euler angle is generated, that is, the Euler angle is not equal to 0, at this time, the quantization value of the posture information corresponding to the facial posture information decreases. The variation range of the quantized value of the attitude information caused by the angle of the Euler angles can be controlled by the variance matrix. Therefore, the variance of the Euler angle can be set to adjust the degree of influence of the Euler angle on the quantized value of the attitude information.

202. Use any video frame in any video as the current video frame;

The operations performed in this step are similar to the operations performed in step 102 in the foregoing embodiment shown in FIG. 1 , and details are not described herein again.

203. Based on the reinforcement learning algorithm, calculate the reward value of the action under the current video frame according to the pose information quantization value of the current video frame and the optical flow energy change value, and determine the candidate video frame corresponding to the maximum reward value as the next video frame of the current video frame. Video frame, take the next video frame of the current video frame as the new current video frame, and return to execute the algorithm based on reinforcement learning, according to the pose information quantization value of the current video frame and the optical flow energy change value to calculate the action under the current video frame. return value steps;

In this embodiment, when calculating the reward value of an action, the specific calculation method is to determine the transition probability from the current video frame to the candidate video frame under the action, and calculate the The initial reward value of the action under the current video frame, and the product of the initial reward value and the transition probability is taken as the reward value of the action.

Specifically, the specific method of determining the transition probability is: when a preset condition is satisfied, the transition probability is determined to be 1; when any of the preset conditions is not satisfied, the transition probability is determined to be 0. The preset conditions include: the current video frame and the candidate video frame are adjacent on the timeline, the target person exists in the picture of the candidate video frame, and the video index corresponding to the action is consistent with the index of the candidate video frame in at least one video.

The fact that the current video frame and the candidate video frame are adjacent on the timeline means that the current video frame and the candidate video frame are adjacent on the video timeline. For example, if the current video frame is the T-th video frame in the first channel video, the candidate video frame may be the T+1-th frame in the current video or other videos (such as the second channel video, the third channel video, etc.) video frame.

204. Determine a video frame sequence according to the sequence of the current video frames, and obtain an initial composite video based on the video frame sequence;

The operations performed in this step are similar to the operations performed in step 104 in the foregoing embodiment shown in FIG. 1 , and details are not described herein again.

205, according to the position and size of each frame of the target character in the initial composite video, determine the position and size of the video picture window of each frame in the initial composite video;

In this embodiment, the position and size of the target character in the video frame can be determined according to the information of the target character and the object interacting with the target character, wherein the information of the object interacting with the target character can be the focus of the sight direction of the target character object information. For example, if the sight direction of the target person focuses on a chair, the determined position and size of the target person in the video frame should include not only the target person, but also the chair that the target person's sight direction focuses on.

Specifically, the specific method of determining the direction of sight of the target person in the video frame may be, since the direction of sight is related to the facial posture information of the target person, for example, it can be determined that the face up is the action of looking up, and the face down can be determined to be the action of looking up. Therefore, the gaze direction of the target person in each frame of the initial composite video can be determined according to the facial posture information of the target person in each frame of the initial composite video.

For example, looking left and right mainly depends on the yaw angle (yaw) in the facial pose information, so the direction of sight of the target person can be determined according to the yaw angle. In order to facilitate the subsequent calculation process, the line of sight direction can be quantified. For example, the line of sight direction can be represented numerically based on the following formula:

Among them, g refers to the line-of-sight direction, and φ refers to the yaw angle. Therefore, the value of the line-of-sight direction corresponding to a certain yaw angle can be determined according to the above formula.

After determining the value of the sight direction, further determine the position and size of each frame of the target person in the initial composite video according to the sight direction of the target person in each frame of the initial composite video, and determine the position and size of each frame of the target person in the initial composite video according to the target person The position and size of each frame in the video determines the position and size of the video picture window of each frame in the initial composite video.

其中，

指代目标人物在初始合成视频的每一帧中的坐标，即根据

和

可以确定目标人物在视频帧中的位置；

指代目标人物在初始合成视频的每一帧中的宽，

指代目标人物在初始合成视频的每一帧中的高，即根据

和

可以确定目标人物在视频帧中的尺寸大小。Specifically, the position and size of each frame of the target character in the initial composite video can be expressed as

in,

Refers to the coordinates of the target person in each frame of the initial composite video, that is, according to

and

The position of the target person in the video frame can be determined;

refers to the width of the target person in each frame of the initial composite video,

Refers to the height of the target person in each frame of the initial composite video, that is, according to

and

The size of the target person in the video frame can be determined.

其中，

指代视频画面窗口在视频帧中的坐标，即根据

可以确定视频画面窗口在初始合成视频中的视频帧的具体位置；

指代视频画面窗口的宽，

指代视频画面窗口的高，即根据

和

可以确定视频画面窗口的尺寸大小。At the same time, the position and size of the video picture window of each frame in the initial composite video can be expressed as

in,

Refers to the coordinates of the video screen window in the video frame, that is, according to

The specific position of the video frame window in the initial composite video can be determined;

Indicates the width of the video screen window,

Refers to the height of the video screen window, that is, according to

and

You can determine the size of the video screen window.

具体的，每一帧的视频画面窗口的位置和尺寸大小可以根据以下目标函数计算得到：In this embodiment, solving according to the function

Specifically, the position and size of the video picture window of each frame can be calculated according to the following objective function:

Among them, ct refers to the initial composite video, _t refers to any video frame in _ct , and _gt refers to the value of the line-of-sight direction mentioned above.

的最优解。Since the above objective function is a convex function, the convex optimization algorithm can be used to solve the objective function, and the optimal position and size of the video picture window of each frame in the initial synthetic video c _t can be solved, that is, to find have to

the optimal solution.

Therefore, according to the expression of the above objective function, it can be seen that when determining the position and size of the video screen window, the information of the objects interacting with the target person in the direction of sight of the target person is also considered, and the interaction of the target person is taken into account. The object is included in the video frame window.

206. Extract the video picture of each frame in the initial composite video based on the position and size of the video picture window to obtain the target composite video;

After the position and size of the video picture window are determined, the video picture of each frame in the initial composite video can be extracted based on the position and size of the video picture window, and the extracted multi-frame video pictures constitute the target composite video. According to the above description, since the position and size of the video picture window are determined based on the target person and the objects interacting with the target person, the video picture extracted according to the video picture window includes the information of the target person and the information of the objects interacting with the target person. information, and avoid the information of other irrelevant characters from being presented in the video screen, on the one hand, it can maximize the presentation of the target character’s information, and on the other hand, it also avoids the presentation of other irrelevant characters’ information, avoiding the problem of privacy leakage.

In this embodiment, the information of the target person is emphasized on the automatic video editing, and the privacy leakage problem is avoided, so that the technical solution has more practical application value, and the practicability of the solution is improved.

The automatic video editing method in the embodiment of the present application has been described above, and the automatic video editing device in the embodiment of the present application is described below. Referring to FIG. 4, an embodiment of the automatic video editing device in the embodiment of the present application includes:

The calculation unit 401 is used to calculate the facial posture information of the target person of each video frame in at least one video, calculate the quantized value of posture information corresponding to the facial posture information, and calculate the optical flow energy of each video frame. change value;

Determining unit 402, for using any video frame in any video as the current video frame;

The editing unit 403 is configured to, based on the reinforcement learning algorithm, calculate the reward value of the action under the current video frame according to the pose information quantization value and the optical flow energy change value of the current video frame, and determine the candidate video frame corresponding to the maximum reward value as The next video frame of the current video frame, taking the next video frame of the current video frame as the new current video frame, and returning to execute the reinforcement learning algorithm based on the quantization value and optical flow according to the pose information of the current video frame The step of calculating the reward value of the action under the current video frame by the energy change value; wherein, the action is to select a candidate video frame from each video of the at least one video respectively;

A generating unit 404, configured to determine a sequence of video frames according to the sequence of the current video frames, and obtain an initial composite video based on the sequence of video frames;

The determining unit 402 is further configured to determine the position and size of the video picture window of each frame in the initial composite video according to the position and size of each frame of the target character in the initial composite video. ;

The extraction unit 405 is configured to extract the video picture of each frame in the initial composite video based on the position and size of the video picture window to obtain a target composite video.

In a preferred implementation of this embodiment, the calculation unit 401 is specifically configured to calculate the facial posture information of the target person of each video frame according to the facial posture estimation algorithm, and the facial posture information includes the angle of the pitch angle , the angle of the yaw angle, and the angle of the rotation angle.

In a preferred implementation manner of this embodiment, the calculating unit 401 is specifically configured to use a multivariate Gaussian model to calculate the quantized value of the posture information corresponding to the facial posture information.

In a preferred implementation of this embodiment, the calculation unit 401 is specifically configured to calculate the optical flow information of each video frame and the optical flow information of other video frames belonging to the same video as each video frame; according to The optical flow information of each video frame calculates the optical flow energy of each video frame, calculates the optical flow energy of the other video frames according to the optical flow information of the other video frames, and calculates the each video frame The optical flow energy difference with the other video frames and the interval time between each video frame and the other video frames; take the quotient of the optical flow energy difference and the interval time as the each video The optical flow energy change value for the frame.

In a preferred implementation of this embodiment, the editing unit 403 is specifically configured to determine the transition probability from the current video frame to the candidate video frame under the action; quantify the value and optical flow according to the pose information of the current video frame The energy change value calculates the initial reward value of the action under the current video frame; the reward value is the product of the initial reward value and the transition probability.

In a preferred implementation of this embodiment, the editing unit 403 is specifically configured to determine that the transition probability is 1 when a preset condition is satisfied; The transition probability is 0;

The preset conditions include: the current video frame and the candidate video frame are adjacent on the timeline, the target person exists in the picture of the candidate video frame, the video index corresponding to the action is the same as The indices of the candidate video frames in the at least one channel of video are consistent.

In a preferred implementation of this embodiment, the determining unit 402 is specifically configured to determine that the target person is in the initial composite video according to the facial posture information of each frame of the target person in the initial composite video The line of sight direction of each frame of the target person; determine the position and size of each frame of the target person in the initial composite video according to the sight direction of the target person in each frame of the initial composite video; According to The position and size of each frame of the target person in the initial composite video determine the position and size of the video picture window of each frame in the initial composite video.

In this embodiment, the operations performed by each unit in the apparatus for automatic video editing are similar to those described in the foregoing embodiments shown in FIG. 1 to FIG. 2 , and details are not repeated here.

In this embodiment, the calculation unit 401 calculates the quantized value of the facial posture information and the optical flow energy change value of the target person in each video frame, and the editing unit 403 applies the calculated quantized value of the posture information and the optical flow energy change value to the enhancement The calculation of the reward value of the action in the learning algorithm, and the candidate video frame corresponding to the maximum reward value is determined as the next video frame of the current video frame, and the next video frame of the current video frame is regarded as the new current video frame, and return to execute The step of calculating the reward value of the action under the current video frame, so that each video frame determined from at least one video can maximize the information of the target person and avoid showing the screen where the target person is blocked. At the same time, the determining unit 402 determines the video picture window based on the position and size of the target person in the video frame, and the extracting unit 405 extracts the video picture about the target person according to the video picture window, so that the final synthesized video maximizes the relevant information about the target person. information about characters, and avoid showing information about other irrelevant characters.

The following describes the automatic video editing device in the embodiment of the present application. Please refer to FIG. 5 . An embodiment of the automatic video editing device in the embodiment of the present application includes:

The automatic video editing apparatus 500 may include one or more central processing units (CPU) 501 and a memory 505, and the memory 505 stores one or more application programs or data.

Among them, the memory 505 may be volatile storage or persistent storage. The program stored in the memory 505 may include one or more modules, each module may include a series of instruction operations in the video automatic editing apparatus. Furthermore, the central processing unit 501 may be configured to communicate with the memory 505 to execute a series of instruction operations in the memory 505 on the automatic video editing apparatus 500 .

Video automatic editing device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and/or, one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

The central processing unit 501 can perform the operations performed by the automatic video editing apparatus in the embodiments shown in FIG. 1 to FIG. 2 , and details are not repeated here.

An embodiment of the present application further provides a computer storage medium, wherein an embodiment includes: an instruction is stored in the computer storage medium, and when the instruction is executed on a computer, causes the computer to execute the foregoing embodiments shown in FIG. 1 to FIG. 2 . The operation performed by the automatic video editing device in .

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above may refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk and other media that can store program codes.

Claims (10)

1. a video automatic editing method, is characterized in that, described method comprises: Calculate the facial posture information of the target person of each video frame in at least one video, calculate the posture information quantization value corresponding to the facial posture information, and calculate the optical flow energy change value of each video frame; Use any video frame in any video as the current video frame; Based on the reinforcement learning algorithm, the reward value of the action under the current video frame is calculated according to the pose information quantization value and the optical flow energy change value of the current video frame, and the candidate video frame corresponding to the maximum reward value is determined as the current video frame. The next video frame, taking the next video frame of the current video frame as the new current video frame, and returning to execute the reinforcement learning algorithm, according to the pose information quantization value of the current video frame and the optical flow energy change value calculated at the current video frame. The step of the reward value of the action under the current video frame; wherein, the action is to select a candidate video frame from each video of the at least one video respectively; Determine a video frame sequence according to the sequence of the current video frames, and obtain an initial composite video based on the video frame sequence; According to the position and size of each frame of the target character in the initial composite video, determine the position and size of the video picture window of each frame in the initial composite video; Based on the position and size of the video picture window, a video picture of each frame in the initial composite video is extracted to obtain a target composite video. 2. The method according to claim 1, wherein the calculating the facial posture information of the target person of each video frame in at least one video, comprises: The facial posture information of the target person of each video frame is calculated according to the facial posture estimation algorithm, and the facial posture information includes the angle of the pitch angle, the angle of the yaw angle and the angle of the rotation angle. 3. The method according to claim 1, wherein the calculating the quantized value of the posture information corresponding to the facial posture information comprises: The quantized value of the posture information corresponding to the facial posture information is calculated by using a multivariate Gaussian model. 4. The method according to claim 1, wherein the calculating the optical flow energy change value of each video frame comprises: calculating the optical flow information of each video frame and the optical flow information of other video frames belonging to the same video as each video frame; Calculate the optical flow energy of each video frame according to the optical flow information of each video frame, calculate the optical flow energy of the other video frames according to the optical flow information of the other video frames, and calculate the each video frame The optical flow energy difference between the frame and the other video frames and the interval time between each video frame and the other video frames; Taking the quotient of the optical flow energy difference and the interval time as the optical flow energy change value of each video frame. 5. The method according to claim 1, wherein calculating the reward value of the action under the current video frame according to the quantized value of the posture information and the optical flow energy change value of the current video frame, comprising: determining the transition probability of the current video frame to the candidate video frame under the action; Calculate the initial reward value of the action under the current video frame according to the pose information quantization value and the optical flow energy change value of the current video frame; The reward value is the product of the initial reward value and the transition probability. 6. The method according to claim 5, wherein the determining the transition probability of the current video frame to the candidate video frame under the action comprises: When a preset condition is met, the transition probability is determined to be 1; when any one of the preset conditions is not met, the transition probability is determined to be 0; The preset conditions include: the current video frame and the candidate video frame are adjacent on the timeline, the target person exists in the picture of the candidate video frame, the video index corresponding to the action is the same as The indices of the candidate video frames in the at least one channel of video are consistent. 7. The method according to claim 1, wherein, determining each frame in the initial composite video according to the position and size of each frame of the target character in the initial composite video The position and size of the video screen window, including: Determine the line-of-sight direction of the target person in each frame of the initial composite video according to the facial posture information of the target person in each frame of the initial composite video; Determine the position and size of each frame of the target character in the initial composite video according to the sight direction of the target character in each frame of the initial composite video; According to the position and size of each frame of the target person in the initial composite video, the position and size of the video picture window of each frame in the initial composite video are determined. 8. An automatic video editing device, wherein the device comprises: A calculation unit, used to calculate the facial posture information of the target person in each video frame in at least one video, calculate the quantized value of posture information corresponding to the facial posture information, and calculate the optical flow energy change of each video frame. value; Determining unit, for taking any video frame in any video as the current video frame; The editing unit is used to calculate the reward value of the action under the current video frame according to the pose information quantization value and the optical flow energy change value of the current video frame based on the reinforcement learning algorithm, and determine the candidate video frame corresponding to the maximum reward value as the selected video frame. Describe the next video frame of the current video frame, take the next video frame of the current video frame as the new current video frame, and return to execute the described based reinforcement learning algorithm, according to the attitude information quantization value and optical flow energy of the current video frame The step of calculating the reward value of the action under the current video frame by the change value; wherein, the action is to select a candidate video frame from each video of the at least one video respectively; a generating unit, configured to determine a sequence of video frames according to the sequence of the current video frames, and obtain an initial composite video based on the sequence of video frames; The determining unit is further configured to determine the position and size of the video picture window of each frame in the initial composite video according to the position and size of each frame of the target character in the initial composite video; An extraction unit, configured to extract the video picture of each frame in the initial composite video based on the position and size of the video picture window to obtain a target composite video. 9. An automatic video editing device, wherein the device comprises: a memory for storing a computer program; a processor for implementing the steps of the automatic video editing method according to any one of claims 1 to 7 when the computer program is executed. 10. A computer storage medium, characterized in that the computer storage medium stores instructions, and when the instructions are executed on a computer, the instructions cause the computer to execute the method according to any one of claims 1 to 7 .

Images