CN111476868B - Animation generation model training and animation generation method and device based on deep learning - Google Patents

Abstract

The invention provides a training of an animation generation model based on deep learning, an animation generation method and a device, relating to the field of computer vision, comprising the following steps: acquiring a training set sequence, wherein the training set sequence comprises a plurality of key frame images; inputting the initial key frame into an animation generation model, and determining a predicted subsequent key frame; determining a value of a loss function according to the predicted subsequent key frame and the actual subsequent key frame; and adjusting parameters of the animation generation model according to the value of the loss function until convergence conditions are met, and completing training of the animation generation model. The invention takes the input as the initial key frame, the initial key frame as the input is given by an animator, other predicted subsequent key frames are supplemented by the trained model, and a series of animations are effectively generated through the model, thereby not only ensuring that the character model designed by the animator is not changed, but also avoiding a great deal of repeated work of the animator.

Description

Animation generation model training, animation generation method and device based on deep learning

technical field

The present invention relates to the field of computer vision, in particular to a deep learning-based animation generation model training, animation generation method and device.

Background technique

Most animations rely on computer keyframing techniques. Computer key frame technology is currently the mainstream method used to complete complicated interpolation of intermediate frames in the field of computer animation. Its basic idea is to interpolate the intermediate frame between two key frames through interpolation. After that, technologies such as linear interpolation, spline interpolation, double interpolation, motion deformation, and motion mapping based on this idea have met people's needs for animation generation to a certain extent.

Most of the methods used so far in the field of computer animation generation are still based on keyframes, but no matter what kind of keyframe acquisition method, there is still manual work. With the rise of deep learning technology in recent years, how to use deep learning technology to allow computers to automatically generate animation has gradually become a new research hotspot in the field of computer animation.

At present, the main application of deep learning technology in the field of computer animation is the method of using the feature points of specific objects combined with deep learning to generate key frames. However, it is difficult to extract the relevant feature points in actual operation. Due to the inaccurate feature point extraction, the generated results Not ideal. In the application of deep learning technology in the field of computer animation, there is also a method of modeling objects in advance for parameterized generation, modeling specific objects, parameterizing the objects, and then changing the parameters over time to achieve the purpose of generating animations, but due to Pixel-style characters are diverse, and most of the results of character model parameterization are not beneficial to the generation of walking animations (such as tall and short, fat and thin, etc.), which simply increases a lot of workload and is different from the actual application method of pixel-style character animation Mismatch.

Contents of the invention

The present invention aims to solve the technical problems in related technologies at least to a certain extent. To achieve the above-mentioned purpose, the embodiment of the first aspect of the present invention provides a deep learning-based animation generation model training method, which includes:

Obtain a training set sequence, the training set sequence includes a plurality of key frame sequences, each of the key frame sequences includes a plurality of key frame images of different walking postures of the characters, and the plurality of key frame images include initial key frames and corresponding The actual subsequent key frame, the initial key frame is the walking posture of the character in the initial state, and the actual subsequent key frame is the walking posture of the character after the initial state;

Inputting the initial key frame into the animation generation model to determine the predicted subsequent key frame;

determining a value of a loss function according to the predicted subsequent key frame and the actual subsequent key frame;

Adjusting the parameters of the animation generation model according to the value of the loss function until a convergence condition is met, and completing the training of the animation generation model.

Therefore, the present invention uses the input as the initial key frame, the initial key frame as the input is given by the animator, and the trained model is used to complete other predicted subsequent key frames, and a series of animations are effectively generated through the model, which ensures This ensures that the character modeling designed by the animator will not be changed, and avoids a lot of repetitive work for the animator.

Further, the walking posture includes the walking direction and walking action of the character, the walking direction includes front, rear, left and right, and the walking action includes left leg stepping action, right leg stepping action and stillness.

Therefore, through the method of image translation, based on the complete action material, the pixel-style characters face different walking directions and make different walking posture training, effectively translating the key frames of the pixel characters facing one direction and stilling into another direction. Keyframes in one direction ensure the accuracy of predicting subsequent keyframes.

Further, the animation generation model includes a first cascade network and a second cascade network, and the predicted subsequent key frames include a first cascaded subsequent key frame and a second cascaded subsequent key frame; The initial key frame is input into the animation generation model, and determining the predicted subsequent key frame includes:

inputting the initial key frame into the first cascade network, and determining the subsequent key frame of the first cascade;

Inputting the first concatenated subsequent key frame to the second concatenated network to determine the second concatenated subsequent key frame.

Therefore, by designing the cascaded network, the present invention can effectively generate a complete task walking action by utilizing the division of labor between the first cascaded network and the second cascaded network. Predicted subsequent keyframes of the pixel character's walking action.

Further, the inputting the initial key frame to the first cascade network includes: inputting the initial key frame to the first cascade network after mirror transformation.

Therefore, the present invention effectively avoids network training difficulties caused by local asymmetry of characters through mirror transformation.

Further, the walking posture includes the walking direction and walking action of the character, the first cascade network includes a first generation subnetwork and a second generation subnetwork, and the first cascade subsequent key frame includes a first generation subnetwork A key frame and a second key frame are generated; the initial key frame is input to the first cascaded network, and the determination of the first cascaded subsequent key frame includes;

Inputting the initial key frame into the first generating sub-network, determining the first generating key frame, the first generating key frame has the same walking direction as the initial key frame, and the walking action is different ;

Inputting the initial key frame into the second generating sub-network, determining the second generating key frame, the second generating key frame is different from the initial key frame in the walking direction, and the walking action is the same .

Thus, in the first cascaded network, the initial keyframes are effectively translated into keyframes facing the same direction through the first generation subnetwork, and the initial keyframes are effectively translated into keyframes facing different directions through the second generation subnetwork. frame.

Further, the walking direction includes a first direction, and the initial key frame is an initial frame of the first direction, which is used to represent the initial state of the character facing the first direction, and the first generated key frame includes a plurality of first direction In the action subframe, the walking actions of the plurality of first direction action subframes are different from each other.

Thus, the initial key frame is effectively translated into key frames for taking different actions facing the first direction through the first generation sub-network.

Further, the second generation sub-network includes a generator and a first discriminator, wherein:

The generator includes a plurality of convolutional layers, and the generator is used to output and generate a prediction map;

The first discriminator includes a plurality of convolutional layers and fully connected layers, and the first discriminator is used to compare the generated prediction map, the initial key frame and the initial key frame in the second generation subnetwork The actual subsequent key frame corresponding to , the output of the first discriminator is the compared discriminant result.

Thus, the generator effectively outputs the generated prediction map, and the discriminator effectively judges whether the generated prediction map is accurate.

Further, the loss function of the generator adopts an L1 loss function, and the first discriminator adopts a cross-entropy loss function.

Therefore, according to the generator and the discriminator, the loss function is selected in a targeted manner to ensure the efficiency and accuracy of network training.

Further, the second cascaded subsequent key frame is a third generated key frame; the input of the first cascaded subsequent key frame to the second cascaded network determines the second cascaded Subsequent keyframes include:

Inputting the second generated key frame into the second cascaded network to determine the third generated key frame, the third generated key frame is the same as the walking direction of the second generated key frame, and the The walking action is different.

Thus, through the second cascaded network, the second generated keyframes are effectively translated into keyframes facing different actions in the same direction.

Further, the walking direction includes a second direction, a third direction, and a fourth direction, and the second generated key frame includes an initial frame in the second direction, an initial frame in the third direction, and an initial frame in the fourth direction, and the second The cascade network includes a third generation subnetwork, a fourth generation subnetwork and a fifth generation subnetwork, the second generation key frame is input to the second cascade network, and the third generation key frame is determined include:

inputting the initial frame in the second direction to the third generating subnetwork, and determining a plurality of action subframes in the second direction, where the action subframe in the second direction is the same as the walking direction of the initial frame in the second direction . The walking motions are different, and the walking motions of the multiple second-direction motion sub-frames are different from each other;

inputting the initial frame of the third direction into the fourth generation subnetwork, and determining a plurality of action subframes of the third direction, the action subframe of the third direction is the same as the walking direction of the initial frame of the third direction . The walking motions are different, and the walking motions of the multiple third-direction motion subframes are different from each other;

inputting the initial frame of the fourth direction into the fifth generation subnetwork, and determining a plurality of action sub-frames of the fourth direction, the action sub-frame of the fourth direction is the same as the walking direction of the initial frame of the fourth direction . The walking motions are different, and the walking motions of the multiple fourth-direction motion sub-frames are different from each other.

Thus, through the third generation sub-network, the initial frames in the second direction are effectively translated into key frames oriented to different actions in the same direction; through the fourth generation sub-network, the initial frames in the third direction are effectively translated into Keyframes for different actions in different directions; through the fifth generation sub-network, the initial frame in the fourth direction is effectively translated into keyframes for different actions in the same direction.

Further, the first generation subnetwork, the third generation subnetwork, the fourth generation subnetwork and the fifth generation subnetwork are the same, including a forward generator, a reverse generator and a second discriminator ,in:

The forward generator includes a plurality of convolutional layers, the input of the forward generator is the input key frame image corresponding to the generated subnetwork, and the output is the forward generated prediction map;

The reverse generator includes a plurality of convolutional layers, the input of the reverse generator is the forward generation prediction map, and the output is the restoration generation prediction map;

The second discriminator includes a plurality of convolutional layers and a fully connected layer, and the second discriminator is used to compare the forward generated prediction map, the input key frame image, and the input of the second discriminator The key frame image corresponds to the actual subsequent key frame in the generating subnetwork, and the output of the second discriminator is a compared discriminant result.

Thus, the forward generator can effectively output the forward generated prediction map, the reverse generator can effectively generate the restored predicted map, and the discriminator can effectively judge whether the generated predicted map is accurate.

Further, the loss function of the forward generator and the loss function of the reverse generator both use the L1 loss function, and the determination process of the loss function of the second discriminator includes:

Determining a first L1 loss function according to the restored prediction map and the input key frame image;

determining a second L1 loss function according to the forward generated prediction map and the actual subsequent key frame corresponding to the input key frame image in the sub-network;

Weighting and summing the first L1 loss function and the second L1 loss function to determine a loss function of the second discriminator.

Therefore, according to the forward generator, reverse generator and discriminator, the loss function is selected in a targeted manner to ensure the efficiency and accuracy of network training.

In order to achieve the above purpose, the embodiment of the second aspect of the present invention also provides a deep learning-based animation generation model training device, which includes:

Acquisition unit: used to obtain a training set sequence, the training set sequence includes a plurality of key frame sequences, each of which includes a plurality of key frame images of different walking postures of the characters, and the plurality of key frame images include An initial key frame and a corresponding actual subsequent key frame, the initial key frame is the walking posture of the character in the initial state, and the actual subsequent key frame is the walking posture of the character after the initial state;

A processing unit: for inputting the initial key frame into the animation generation model, and determining a predicted subsequent key frame; and for determining a value of a loss function according to the predicted subsequent key frame and the actual subsequent key frame;

A training unit: used to adjust the parameters of the animation generation model according to the value of the loss function until a convergence condition is satisfied, and complete the training of the animation generation model.

The beneficial effects of the deep learning-based animation generation model training device and the deep learning-based animation generation model training method are the same as those of the prior art, and will not be repeated here.

In order to achieve the above purpose, the embodiment of the third aspect of the present invention provides a deep learning-based animation generation method, which includes:

Get the initial state keyframe;

The initial state key frame is input to the animation generation model, and the key frame of the predicted sequence is determined, and the animation generation model is trained by the above-mentioned animation generation model training method based on deep learning;

Carrying out foreground and background segmentation on the initial state key frame and the predicted subsequent key frame, and determining the segmentation result picture;

The segmented result picture is played in a loop to generate an animation.

Thus, the present invention uses the input as the initial key frame, the initial key frame as the input is given by the animator, and the trained model is used to complete other predicted subsequent key frames, and the initial state key frame and the predicted subsequent key frame The key frame is used to segment the foreground and background, so as to accurately obtain the animation of the character walking. To sum up, the present invention effectively generates a series of animations through the model, which not only ensures that the character modeling designed by the animator will not be changed, but also avoids a lot of repeated work by the animator.

Further, the determination of the segmentation result picture includes:

Determine the transparency channel according to the result of the foreground and background segmentation;

According to the transparency channel, the initial state key frame and the predicted subsequent key frame are saved as segmentation result pictures.

Thus, through the transparent channel, the foreground part and the background part are effectively identified, so as to efficiently determine the segmentation result picture.

Further, according to the result of the foreground and background segmentation, determining the transparency channel includes:

For the foreground part, the transparent channel value is 1; for the background part, the transparent channel value is 0.

Therefore, by binarizing the foreground part and the background part, the foreground part and the background part are effectively identified, so as to efficiently determine the segmentation result picture.

Further, according to the transparency channel, saving the initial state key frame and the predicted subsequent key frame as a segmentation result picture includes:

In the initial state key frame and the predicted subsequent key frame, the color of the part whose transparent channel value is 1 is colorless, and the color of the part whose transparent channel value is 0 remains unchanged;

Save the segmentation result image in png format.

Thus, the initial state keyframe and the predicted subsequent keyframe are saved as segmentation result pictures effectively through the transparency channel.

Further, the cyclically playing the segmented result picture to generate animation includes looping for a single playback, and the single playback includes:

Play the segmentation result picture corresponding to the key frame of the initial state;

According to the generation order of the predicted subsequent key frames, the segmentation result pictures corresponding to the predicted subsequent key frames are played sequentially.

Thus, through the generation order of the key frames, the segmentation result picture is effectively played in a loop, so as to generate the corresponding animation.

In order to achieve the above purpose, the embodiment of the fourth aspect of the present invention provides a deep learning-based animation generation device, which includes:

Acquisition unit: used to obtain the initial state keyframe;

Processing unit: for inputting the key frame of the initial state into the animation generation model, and determining the predicted subsequent key frame, and the animation generation model is obtained by training the animation generation model training method based on deep learning as described above; for Perform foreground and background segmentation on the initial state key frame and the predicted subsequent key frame, and save the segmentation result picture;

Playing unit: used for cyclically playing the segmented result picture to generate animation.

The animation generation device based on deep learning provided by the present invention is input as an initial key frame, and the initial key frame as input is given by an animator, and other predicted subsequent key frames are completed by the trained model, and the initial state The key frame and the predicted subsequent key frame are used for foreground and background segmentation, so as to accurately obtain the animation of the character walking. To sum up, the present invention effectively generates a series of animations through the model, which not only ensures that the character modeling designed by the animator will not be changed, but also avoids a lot of repeated work by the animator.

To achieve the above object, the embodiment of the fifth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned animation based on deep learning is realized. Generate a model training method, or implement a deep learning-based animation generation method as described above.

The computer-readable storage medium has the same beneficial effect as the above-mentioned animation generation model training method based on deep learning and the animation generation method based on deep learning relative to the prior art, and will not be repeated here.

Description of drawings

Fig. 1 is the schematic flow chart of the animation generation model training method based on deep learning in the embodiment of the present invention;

Fig. 2 is the specific structural representation of animation generation model in the embodiment of the present invention;

FIG. 3 is a schematic diagram of specific network structures of the first subnetwork, the second subnetwork and the third subnetwork in the embodiment of the present invention;

4 is a schematic diagram of specific network structures from the fourth subnetwork to the eleventh subnetwork in the embodiment of the present invention;

Fig. 5 is a schematic structural diagram of a generator, a forward generator and a reverse generator in an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a discriminator in an embodiment of the present invention;

7 is a schematic structural diagram of an animation generation model training device based on deep learning in an embodiment of the present invention;

FIG. 8 is a schematic flow diagram of a deep learning-based animation generation method according to an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of an animation generation device based on deep learning according to an embodiment of the present invention.

Detailed ways

Embodiments according to the present invention will be described in detail below with reference to the drawings. When the description refers to the drawings, the same reference numerals in different drawings indicate the same or similar elements unless otherwise indicated. It should be noted that the implementations described in the following exemplary embodiments do not represent all implementations of the present invention. They are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as recited in the claims, and the scope of the present invention is not limited thereto. On the premise of no contradiction, the features in the various embodiments of the present invention can be combined with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

Most of the methods used in the field of computer animation generation are still based on keyframes, but no matter what kind of keyframes are obtained, a lot of manual work is required. The use of deep learning technology to automatically generate animations by computers has gradually become a new research hotspot in the field of computer animation. At present, there are mainly two methods for the application of deep learning technology in the field of computer animation. One is to use the feature points of specific objects combined with deep learning to generate key frames. However, it is difficult to extract relevant feature points in actual operation. The inaccurate extraction leads to unsatisfactory generated results; the other is to model the object in advance for parameterized generation, model a specific object, parameterize the object, and then change the parameters over time to achieve The purpose of generating animation, but due to the diversity of pixel-style characters, and the results of parameterizing the model of the characters are mostly not beneficial to the generation of walking animations (such as tall and short, fat and thin, etc.), adding a lot of workload for no reason and pixels The actual application method of style character animation does not match. Therefore, how to establish an animation generation model that is accurate and saves labor costs is an urgent problem to be solved.

An embodiment of the present invention provides a method for training an animation generation model based on deep learning. FIG. 1 is a schematic flow diagram of a deep learning-based animation generation model training method according to an embodiment of the present invention, including steps S101 to S104, wherein:

In step S101, the training set sequence is obtained, the training set sequence includes a plurality of key frame sequences, each key frame sequence includes a plurality of key frame images of different walking postures of the characters, and the plurality of key frame images include initial key frames and corresponding The actual subsequent key frames, the initial key frame is the walking posture of the character in the initial state, and the actual subsequent key frame is the walking posture of the character after the initial state. Thus, the training set sequence is effectively obtained for subsequent network training.

In step S102, the initial key frame is input into the animation generation model, and the predicted subsequent key frame is determined. Thus, the predicted subsequent key frames are generated by the animation generation model.

In step S103, the value of the loss function is determined according to the predicted subsequent key frame and the actual subsequent key frame. Thus, the network is effectively trained by determining the loss function.

In step S104, the parameters of the animation generation model are adjusted according to the value of the loss function until the convergence condition is satisfied, and the training of the animation generation model is completed. Therefore, the present invention uses the input as the initial key frame, the initial key frame as the input is given by the animator, and the trained model is used to complete other predicted subsequent key frames, and a series of animations are effectively generated through the model, which ensures This ensures that the character modeling designed by the animator will not be changed, and avoids a lot of repetitive work for the animator.

Optionally, the walking posture includes the walking direction and walking action of the character, the walking direction includes front, rear, left and right, and the walking action includes left leg stepping action, right leg stepping action and stillness. Therefore, through the method of image translation, based on the complete action material, the pixel-style characters face different walking directions and make different walking posture training, effectively translating the key frames of the pixel characters facing one direction and stilling into another direction. Keyframes in one direction ensure the accuracy of predicting subsequent keyframes.

Specifically, the specific content of the key frame image is as follows: (1) the key frame of the pixel style character facing forward and still; (2) the key frame of the pixel style character facing forward and stepping with his left leg; (3) the pixel style character facing forward , the key frame of the right leg stepping forward; (4) the key frame of the pixel style character facing the rear and standing still; (5) the key frame of the pixel style character facing the rear and stepping the left leg; (6) the pixel style character facing the rear, right The key frame of the leg stepping; (7) the key frame of the pixel style character facing the left and still; (8) the key frame of the pixel style character facing the left and the left leg stepping; (9) the pixel style character facing the left, The key frame of the right leg stepping; (10) the key frame of the pixel style character facing the right and still; (11) the key frame of the pixel style character facing the right and stepping the left leg; (12) the pixel style character facing the right , the key frame of the right leg stepping forward. Therefore, the specific content includes that the pixel-style characters face different walking directions and make different walking postures, which is conducive to translating the key frames of the pixel characters facing one direction into the key frames facing the other direction, or converting the pixel characters Keyframes for standing still facing one direction are translated into keyframes for walking motion facing the same direction.

Optionally, the animation generation model includes a first cascade network and a second cascade network, and the predicted subsequent key frames include the first cascaded subsequent key frames and the second cascaded subsequent key frames. Step S102 includes the following two steps:

The initial key frame is input to the first cascaded network to obtain the first cascaded subsequent key frame. Therefore, the present invention utilizes the first cascaded network to effectively generate the key frames after the first cascaded sequence by designing the cascaded network;

The first cascaded subsequent key frame is input to the second cascaded network to obtain the second cascaded subsequent key frame. Therefore, by designing the cascaded network, the present invention can effectively generate a complete task walking action by utilizing the division of labor between the first cascaded network and the second cascaded network. Predicted subsequent keyframes of the pixel character's walking action.

Optionally, inputting the initial key frame to the first cascaded network includes: inputting the initial key frame to the first cascaded network after mirror transformation. Therefore, the present invention effectively avoids network training difficulties caused by local asymmetry of characters through mirror transformation.

In the embodiment of the present invention, the walking posture includes the walking direction and walking action of the character, and the first cascaded network includes a first generating sub-network and a second generating sub-network. The above inputting the initial key frame to the first cascade network includes the following two steps:

The initial key frame is input to the first generation sub-network, and the first generated key frame is determined. The walking direction of the first generated key frame is the same as that of the initial key frame, and the walking action is different. Thus, in the first cascaded network, the initial key frame is effectively translated into key frames facing the same direction through the first generation sub-network;

The initial key frame is input to the second generation sub-network to determine the second generated key frame. The second generated key frame has a different walking direction and the same walking action as the initial key frame. Thus, in the first cascaded network, the initial keyframes are effectively translated into keyframes facing different directions through the second generation subnetwork.

Optionally, the walking direction includes the first direction, and the initial key frame is the initial frame of the first direction, which is used to represent the initial state of the character facing the first direction. The walking motions of the first direction motion sub-frames are different from each other. Thus, the initial key frame is effectively translated into key frames for taking different actions facing the first direction through the first generation sub-network.

Optionally, the second generation sub-network includes a generator and a first discriminator, wherein: the generator includes multiple convolutional layers for outputting a generated prediction map; the first discriminator includes multiple convolutional layers and a fully connected layer , the first discriminator is used to compare the generated prediction map, the initial key frame, and the actual subsequent key frame corresponding to the initial key frame in the second generation subnetwork, and the output of the first discriminator is the compared discriminant result. Thus, the generator effectively outputs the generated prediction map, and the first discriminator effectively judges whether the generated prediction map is accurate.

Optionally, during the training process, the loss function of the generator adopts the L1 loss function, and the first discriminator adopts the cross-entropy loss function. Therefore, according to the generator and the discriminator, the loss function is selected in a targeted manner to ensure the efficiency and accuracy of network training.

In the embodiment of the present invention, the second cascaded subsequent key frame is the third generated key frame. The above-mentioned inputting the key frame after the first cascade into the second cascade network specifically includes: inputting the second generated key frame into the second cascaded network, determining the third generated key frame, the third generated key frame and the second generated key frame The keyframes have the same walking direction and different walking actions. Thus, through the second cascaded network, the second generated keyframes are effectively translated into keyframes facing different actions in the same direction.

In the embodiment of the present invention, the walking direction includes the second direction, the third direction and the fourth direction, the second generated key frame includes the initial frame of the second direction, the initial frame of the third direction and the initial frame of the fourth direction, and the second cascade The network includes a third generation sub-network, a fourth generation sub-network and a fifth generation sub-network, and the input of the first cascaded subsequent key frames to the second cascade network specifically includes the following three steps:

Input the initial frame of the second direction to the third generating subnetwork to determine a plurality of second direction action subframes, the second direction action subframe and the second direction initial frame have the same walking direction and different walking actions, and the multiple second direction The walking motions of the motion subframes are different from each other. Thus, through the third generation sub-network, the initial frame in the second direction is effectively translated into key frames facing different actions in the same direction;

Input the initial frame of the third direction to the fourth generation subnetwork, determine multiple third-direction action sub-frames, the walking direction of the third-direction action sub-frame and the third-direction initial frame are the same, the walking action is different, and the multiple third-direction The walking motions of the motion sub-frames are different from each other. Thus, through the fourth generation sub-network, the initial frame in the third direction is effectively translated into key frames facing different actions in the same direction;

Input the initial frame of the fourth direction to the fifth generation subnetwork, determine multiple fourth direction action subframes, the fourth direction action subframe and the fourth direction initial frame have the same walking direction and different walking actions, and multiple fourth direction action subframes The walking motions of the motion subframes are different from each other. Thus, through the fifth generation sub-network, the initial frame in the fourth direction is effectively translated into key frames facing different actions in the same direction.

Optionally, the first generation subnetwork, the third generation subnetwork, the fourth generation subnetwork and the fifth generation subnetwork are the same, including a forward generator, a reverse generator and a second discriminator, wherein:

The forward generator includes multiple convolutional layers, the input of the forward generator is the input key frame image corresponding to the generated subnetwork, and the output of the forward generator is the forward generated prediction map;

The reverse generator includes multiple convolutional layers, the input of the reverse generator is to generate a prediction map in the forward direction, and the output of the reverse generator is to restore and generate a prediction map;

The second discriminator includes multiple convolutional layers and fully connected layers. The second discriminator is used to compare the forward generated prediction map, the input key frame image, and the corresponding actual value of the input key frame image of the second discriminator in the generated subnetwork. Subsequent key frames, the output is the compared discriminant result. Thus, the forward generator can effectively output the forward generated prediction map, the reverse generator can effectively generate the restored predicted map, and the second discriminator can effectively judge whether the generated predicted map is accurate.

Optionally, the loss function of the forward generator and the loss function of the reverse generator both use the L1 loss function, and the loss function of the second discriminator is determined by the following steps: determine the first L1 according to the restored prediction map and the input key frame image Loss function; determine the second L1 loss function according to the actual subsequent key frame corresponding to the forward generated prediction map and the input key frame image in the sub-network; weight and sum the first L1 loss function and the second L1 loss function to determine Loss function for the second discriminator. Therefore, according to the forward generator, reverse generator and discriminator, the loss function is selected in a targeted manner to ensure the efficiency and accuracy of network training.

Specifically, referring to FIG. 2 , FIG. 2 is a schematic structural diagram of an animation generation model based on deep learning according to an embodiment of the present invention. Optionally, the predicted subsequent key frames include the first predicted subsequent key frame to the eleventh predicted subsequent key frame. In FIG. 2 , the key frame (2) is the first predicted subsequent key frame, and the key frame ( 3) is the second prediction subsequent key frame, key frame (4) is the third prediction subsequent key frame, key frame (5) is the fourth prediction subsequent key frame, and key frame (6) is the fifth prediction subsequent key frame Key frame, key frame (7) is the sixth prediction subsequent key frame, key frame (8) is the seventh prediction subsequent key frame, key frame (9) is the eighth prediction subsequent key frame, key frame (10) is the ninth predicted subsequent key frame, the key frame (11) is the tenth predicted subsequent key frame, and the key frame (12) is the eleventh predicted subsequent key frame.

In conjunction with Figure 2, the first cascaded network includes a first subnet, a second subnet, a third subnet, a fourth subnet, and a fifth subnet, and the second cascaded network includes a sixth subnet, a seventh subnet, and a fifth subnet. Sub-networks, the eighth sub-network, the ninth sub-network, the tenth sub-network and the eleventh sub-network, wherein the first generation network includes the fourth sub-network and the fifth sub-network, and the second generation network includes the first sub-network , the second subnetwork, the third subnetwork, the third generation network includes the sixth subnetwork, the seventh subnetwork, the fourth generation network includes the eighth subnetwork, the ninth subnetwork, and the fifth generation network includes the tenth subnetwork and the eleventh subnetwork.

In combination with Figure 2, the output of the first subnetwork is the third predicted subsequent keyframe; the output of the second subnetwork is the sixth predicted subsequent keyframe; the output of the third subnetwork is the ninth predicted subsequent keyframe ; The output of the fourth sub-network is the first cascaded post-sequence key frame; the output of the fifth sub-network is the second cascaded post-sequence key frame.

Optionally, in the first cascaded network, the initial key frame is input to the first sub-network to obtain the third predicted subsequent key frame, and the initial key frame is input to the second sub-network to obtain the sixth predicted subsequent key frame, input the initial key frame to the third sub-network to obtain the ninth predicted subsequent key frame, input the initial key frame to the fourth sub-network to obtain the first predicted subsequent key frame, and input the initial key frame to the fifth A sub-network is used to obtain the second predicted subsequent key frame;

Optionally, in the second cascaded network, the third predicted subsequent key frame is input to the sixth sub-network, and the fourth predicted subsequent key frame is obtained as an output; the third predicted subsequent key frame is input to the seventh The sub-network obtains the fifth predicted subsequent key frame as an output; the sixth predicted subsequent key frame is input to the eighth sub-network, and the seventh predicted subsequent key frame is obtained as an output; the sixth predicted subsequent key frame is input to The ninth sub-network obtains the eighth predicted post-sequence key frame as an output; the ninth predicted post-sequence key frame is input to the tenth sub-network, and the tenth predicted post-sequence key frame is obtained as an output; the ninth predicted post-sequence key frame is input to the eleventh sub-network, and the eleventh predicted subsequent key frame is obtained as an output.

Specifically, in conjunction with Figure 2, in the eleven sub-networks:

The first sub-network: the input is the key frame of the pixel-style character facing forward and still, and the output is the key frame of the pixel-style character facing backward and still;

The second sub-network: the input is the key frame of the pixel-style character facing forward and still, and the output is the key frame of the pixel-style character facing left and still;

The third sub-network: the input is the key frame of the pixel-style character facing forward and still, and the output is the key frame of the pixel-style character facing the right and still;

The fourth sub-network: the input is the key frame of the pixel-style character facing forward and still, and the output is the key frame of the pixel-style character facing forward and stepping with the left leg;

The fifth sub-network: the input is the key frame of the pixel-style character facing forward and still, and the output is the key frame of the pixel-style character facing forward and stepping with the right leg;

The sixth sub-network: the input is the key frame of the pixel-style character facing the rear and still, and the output is the key frame of the pixel-style character facing the rear and stepping with the left leg;

The seventh sub-network: the input is the key frame of the pixel-style character facing the rear and still, and the output is the key frame of the pixel-style character facing the rear and stepping with the right leg;

The eighth sub-network: the input is the key frame of the pixel-style character facing the left and still, and the output is the key frame of the pixel-style character facing the left and stepping with the left leg;

The ninth sub-network: the input is the key frame of the pixel-style character facing left and still, and the output is the key frame of the pixel-style character facing the left and stepping on the right leg;

The tenth sub-network: the input is the key frame of the pixel-style character facing the right and still, and the output is the key frame of the pixel-style character facing the right and stepping with the left leg;

The eleventh sub-network: the input is the key frame of the pixel-style character facing the right and standing still, and the output is the key frame of the pixel-style character facing the right and stepping with the right leg.

Specifically, referring to FIG. 3 , FIG. 3 shows a schematic diagram of a specific network structure of the first subnetwork, the second subnetwork and the third subnetwork, including a generator and a discriminator. It can be understood that the role of the network structure in the dotted box is only in training, and does not participate in the animation generation process in actual application.

Specifically, referring to FIG. 4 , FIG. 4 is a schematic diagram of specific network structures from the fourth subnetwork to the eleventh subnetwork, including a forward generator, a reverse generator, and a discriminator. It can be understood that the role of the network structure in the dotted box is only in training, and does not participate in the animation generation process in actual application.

Specifically, referring to FIG. 5 , FIG. 5 is a schematic structural diagram of a generator, a forward generator and a reverse generator according to an embodiment of the present invention. Specifically, it includes 3 downsampling convolution layers, 3 upsampling convolution layers and 1 convolution output layer. The size of the convolution kernel is 3*3 except for the convolution output layer, and the convolution kernel size of the convolution output layer is 1*1; where each downsampling convolutional layer uses 2*2 maximum pooling, and each upsampling convolutional layer is a 2*2 step size convolution. The size of the picture input by each sub-network is 32*32 pixels; the specific network structure of the network is as follows:

Downsampling convolutional layer: first downsampling convolutional layer: 32 convolution kernels, output 16*16*32 feature map; second downsampling convolutional layer: 64 convolution kernels, output 8*8*64 The feature map; the third downsampling convolutional layer: 128 convolution kernels, outputting a 4*4*128 feature map.

Upsampling convolutional layer: first upsampling convolutional layer: 64 convolution kernels, output feature map of 8*8*64; second upsampling convolutional layer: 32 convolution kernels, output 16*16*32 The feature map; the third upsampling convolutional layer: 16 convolution kernels, outputting a feature map of 32*32*16.

Convolution output layer: 3 convolution kernels, outputting a 32*32*3 color output image.

Among them, the downsampling convolutional layer is responsible for extracting the features of the input key frame, and the upsampling convolutional layer is responsible for generating the feature map of the corresponding output key frame from the features extracted by the downsampling convolutional layer, and finally a 1*1 convolutional layer Stable output results.

In the output of each of the above layers, the Relu layer is activated as the output of the layer. The Relu layer is a nonlinear change, and its function expression is as follows:

In the downsampling process, the input of each convolutional layer is the output of the previous convolutional layer. In the upsampling process, the input of the upsampling convolutional layer 1 is the output of the downsampling convolutional layer 3; the upsampling convolutional layer 1 The output of the upsampling convolutional layer 2 and the output of the downsampling convolutional layer 2 are combined and input to the upsampling convolutional layer 2 (as shown by the dotted line in Figure 3), and the output of the upsampling convolutional layer 2 and the output of the downsampling convolutional layer 1 are combined and input The upsampling convolutional layer 3 (as shown by the dotted line in FIG. 3 ), the output of the upsampling convolutional layer 3 enters the final convolutional output layer to output the final result.

Optionally, in the network structure of the generator, the forward generator and the reverse generator, the output of each layer is activated by the Relu layer before being used as the output of the layer. The function and function of the Relu layer are the same as above. The input of the first four layers is the output of the previous level (the first layer is the network input), the input of the fifth layer is the output of the fourth layer, and each pixel is used as a dimension in the fifth layer, which is passed through the fifth layer The fully connected layer outputs the judgment result.

Specifically, referring to FIG. 6 , FIG. 6 is a schematic structural diagram of a discriminator according to an embodiment of the present invention. The discriminator includes the first discriminator and the second discriminator mentioned above. Among them, the size of the convolution kernel in the discriminator is 2*2, and 0 is filled around the feature map for each convolution to ensure that the size of the output feature map is consistent. The discriminator adopts the following structure: the first convolutional layer: 64 convolutional kernels, outputting a feature map of 32*32*64; the second convolutional layer: 128 convolutional kernels, outputting a feature map of 32*32*128 ;Third convolutional layer: 256 convolution kernels, output feature map of 32*32*256; fourth convolution layer: 512 convolution kernels, output feature map of 32*32*512; fifth fully connected layer : The input is 524*288 dimensions, the output is a fully connected layer of 1 dimension, and the judgment result is output.

In the embodiment of the present invention, at the beginning of training, a Gaussian distribution with a mean value of 0 and a variance of 0.1 is used to initialize the network parameters in FIG. 3 . The optimizer uses the Adam optimizer, the learning rate is 0.0005, and the label is the corresponding real key frame in the training set; the training adopts the method of end-to-end training for the sub-networks respectively.

In the embodiment of the present invention, when training the first sub-network, the second sub-network and the third sub-network, the generator loss function uses the L1 loss function, and the discriminator uses the cross-entropy loss function.

The L1 loss function is specifically:

代表子网络的生成结果中第i个点的像素值，y⁽ⁱ⁾代表真实标签中的第i个点的像素值，m为总像素点个数。in

Represents the pixel value of the i-th point in the generated result of the sub-network, y ⁽ⁱ⁾ represents the pixel value of the i-th point in the real label, and m is the total number of pixel points.

The cross-entropy loss function is specifically:

为判别器的输出。Among them, if the picture input to the discriminator is a real picture, y is 1; if it is a picture generated by the generator, y is 0,

is the output of the discriminator.

In the embodiment of the present invention, the discriminator also uses cross-entropy loss when training the fourth to eleventh sub-networks, and the generator loss function uses the L1 loss function of the label image and the L1 loss function of the restored image and the input image The weighted loss function is:

为正向生成器所生成的图片，/>

为逆向生成器所生成的图片，y_l为标签图片，y_i为输入图片。其中λ₁＝1，λ₂＝0.03，所述L1损失函数计算方法同上所述。in

The image generated by the forward generator, />

is the image generated by the reverse generator, y _l is the label image, and y _i is the input image. Where λ ₁ =1, λ ₂ =0.03, and the calculation method of the L1 loss function is the same as that described above.

In the embodiment of the present invention, during the training process, the generator is trained once, and the discriminator is trained 5 times; the training process has a total of 500 cycles, and one cycle means that all the data in the data set are trained once.

In another embodiment of the present invention, for the specific network structure in FIG. 2 , the input of the second sub-network and the third sub-network can be the initial key frame and the output of the first sub-network. Other steps and parameters are the same as above. This change is due to the lack of information on the back of the character when the character is facing forward. For example, only for the key frame of the front of the character, the second sub-network and the third sub-network do not know the information on the back of the character. The key frames of the pixel character facing the front and the rear are input into the network to complete the unknown information of the character on the back, and prevent the key frames of the pixel character facing the left and right from conflicting with the key frames of the pixel character facing the front and rear.

Core(TM)i7 6700HQ，显卡为NVIDIA GTX 1060m 6GB，内存为8GB。由此，进行有效的网络训练。Optionally, the computer can adopt the following configuration: the processor is

Core(TM) i7 6700HQ, graphics card is NVIDIA GTX 1060m 6GB, memory is 8GB. Thus, efficient network training is performed.

The deep learning-based animation generation model training method provided by the present invention uses the input as the initial key frame, which is given by the animator, and the trained model completes other predicted subsequent key frames, through The model effectively generates a series of animations, which not only ensures that the character modeling designed by the animator will not be changed, but also avoids a lot of repetitive work for the animator.

The embodiment of the second aspect of the present invention also provides an animation generation model training device based on deep learning. FIG. 7 is a schematic structural diagram of an animation generation model training device 800 based on deep learning according to an embodiment of the present invention, including an acquisition unit 801 , a processing unit 802 and a training unit 803 .

Acquisition unit 801: used to obtain a training set sequence, the training set sequence includes multiple key frame sequences, each key frame sequence includes multiple key frame images of different walking postures of the characters, and the multiple key frame images include initial key frames and corresponding The actual subsequent key frames of , the initial key frame is the walking posture of the character in the initial state, and the actual subsequent key frame is the walking posture of the character after the initial state;

Processing unit 802: used to input the initial key frame into the animation generation model, and determine the predicted subsequent key frame; also used to determine the value of the loss function according to the predicted subsequent key frame and the actual subsequent key frame;

Training unit 803: used to adjust the parameters of the animation generation model according to the value of the loss function until the convergence condition is satisfied, and complete the training of the animation generation model.

For a more specific implementation of each unit of the deep learning-based animation generation model training device 800, please refer to the description of the deep learning-based animation generation model training method of the present invention, and have similar beneficial effects, which will not be discussed here. Let me repeat.

The embodiment of the third aspect of the present invention proposes a deep learning-based animation generation method. FIG. 8 is a schematic flowchart of a method for generating animation based on deep learning according to an embodiment of the present invention, including steps S201 to S204.

In step S201, an initial state key frame is obtained. In the present invention, the input is an initial key frame, and the initial key frame as input is given by an animator.

In step S202, the initial state key frame is input into the animation generation model, and the predicted subsequent key frame is determined, and the animation generation model is trained by the above-mentioned deep learning-based animation generation model training method. As a result, other predicted subsequent key frames are completed by the trained model.

In step S203, perform foreground and background segmentation on the initial state key frame and the predicted subsequent key frame, and determine the segmentation result picture. Therefore, the foreground and background segmentation is performed on the initial state key frame and the predicted subsequent key frame, so as to accurately obtain the animation of the character walking.

In step S204, the segmentation result picture is played in a loop to generate an animation. Therefore, the present invention effectively generates a series of animations through the model, which not only ensures that the character modeling designed by the animator will not be changed, but also avoids a lot of repeated work by the animator.

In the embodiment of the present invention, step S203 includes the following two specific steps:

Determine the transparency channel according to the result of foreground and background segmentation;

According to the transparency channel, the initial state keyframe and the predicted subsequent keyframe are saved as the segmentation result image. Thus, through the transparent channel, the foreground part and the background part are effectively identified, so as to efficiently determine the segmentation result picture.

Optionally, for the foreground part, the transparent channel value is 1; for the background part, the transparent channel value is 0. Therefore, by binarizing the foreground part and the background part, the foreground part and the background part are effectively identified, so as to efficiently determine the segmentation result picture.

Optionally, in the initial state keyframe and the predicted subsequent keyframe, the color of the part with a transparent channel value of 1 is colorless, and the color of the part with a transparent channel value of 0 remains unchanged; save the segmentation in png format Result image. Thus, the initial state keyframe and the predicted subsequent keyframe are saved as segmentation result pictures effectively through the transparency channel. Because the generated picture is in RGB format, that is, it corresponds to a pixel point, and the color of the point is represented by three components of red, green and blue. If there is no segmentation, for the 32*32 character key frame, the parts around the character have colors , so that when the animation is generated, the generated result will also carry the surrounding color. This part of the color is not needed by us, so it needs to be divided, and the unnecessary color background around the generated character must be cleared. The generated keyframe needs to be saved in RGBA format, which is similar to the above-mentioned RGB format picture, corresponding to the same pixel, and the color of the point is represented by three components of red, green and blue, and a transparency channel indicates the transparency of the pixel, as long as the transparency If the channel is 1, it means that the color represented by the pixel is "colorless", and if the transparency channel is 0, it means the color determined by the three components of red, green and blue.

Optionally, the region growing method is used for foreground and background segmentation, and the upper left corner pixel of each generated key frame is used as the seed point of the background, and the region growing method is used to calculate the color gradient on the boundary of the background region, and if it is less than the threshold, it will be Merge into the background area and execute repeatedly until the area no longer grows. Taking the specific animation generation model in Figure 2 as an example, the initial key frame of the pixel-style character walking animation and the pixel points in the upper left corner of the other 11 predicted subsequent key frames output by the model are used as seed points, and the initial key frame is divided by region growing. The image foreground and image background of the key frame and the other 11 predicted subsequent key frames output by the model.

In the embodiment of the present invention, loop playback includes loop single playback, and the step of single playback includes step S204 including the following two specific steps:

Play the segmentation result picture corresponding to the key frame of the initial state;

According to the generation sequence of the predicted key frames, the segmentation result pictures corresponding to the predicted key frames are played in sequence. Thus, through the generation order of the key frames, the segmentation result picture is effectively played in a loop, so as to generate the corresponding animation. Take the specific animation generation model in Figure 2 as an example, loop through the png images of 12 key frames of the character, and play each key frame in a loop to generate a pixel-style character walking animation; the generated key frame can also be interpolated by key frame interpolation. Frames are interpolated to produce finer animations.

The deep learning-based animation generation method provided by the present invention uses the input as the initial key frame, which is given by the animator, and the trained model is used to complete other predicted subsequent key frames, and the model is effective Generating a series of animations not only ensures that the character modeling designed by the animator will not be changed, but also avoids a lot of repetitive work for the animator.

The embodiment of the fourth aspect of the present invention proposes an animation generation device based on deep learning. FIG. 9 is a schematic structural diagram of an animation generation device 900 based on deep learning according to an embodiment of the present invention, including an acquisition unit 901 , a processing unit 902 and a playback unit 903 .

The obtaining unit 901 is configured to obtain an initial state key frame.

The processing unit 902 is configured to input the initial state key frame into the animation generation model, and determine the predicted subsequent key frame, and the animation generation model is obtained by training the animation generation model training method based on deep learning as described above; for the initial state Foreground and background segmentation is performed on the key frame and the predicted key frame, and the segmentation result image is saved.

The playback unit 903 is configured to play the segmented result picture in a loop to generate animation.

For a more specific implementation of each unit of the deep learning-based animation generation device 900 , refer to the description of the deep learning-based animation generation method of the present invention, which has similar beneficial effects and will not be repeated here.

The embodiment of the fifth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the deep learning-based animation generation model according to the first aspect of the present invention is realized. Training method, or realize the animation generation method based on deep learning according to the third aspect of the present invention.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and cannot be construed as limiting the present invention, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (19)

1. An animation generation model training method based on deep learning, which is characterized by comprising the following steps:

acquiring a training set sequence, wherein the training set sequence comprises a plurality of key frame sequences, each key frame sequence comprises a plurality of key frame images of different walking postures of a person, each key frame image comprises an initial key frame and a corresponding actual subsequent key frame, the initial key frame is the walking posture of the person in an initial state, and the actual subsequent key frame is the walking posture of the person after the initial state; the walking gesture comprises a walking direction and a walking action of the character;

Inputting the initial key frame into the animation generation model, and determining a predicted subsequent key frame;

the animation generation model comprises a first cascade network and a second cascade network, and the predicted subsequent key frames comprise a first cascade subsequent key frame and a second cascade subsequent key frame; inputting the initial key frame into the first cascade network, and determining the first cascade subsequent key frame; inputting the first cascade subsequent key frame into the second cascade network, and determining the second cascade subsequent key frame;

the first cascade subsequent key frame comprises a first generated key frame and a second generated key frame; the second cascade subsequent key frame is a third generated key frame;

the first generated key frame is the same as the walking direction of the initial key frame and the walking action is different; the second generated key frame is different from the initial key frame in the walking direction and the walking action is the same; the walking direction of the third generated key frame is the same as that of the second generated key frame, and the walking actions are different;

determining a value of a loss function from the predicted subsequent key frame and the actual subsequent key frame;

And adjusting parameters of the animation generation model according to the value of the loss function until convergence conditions are met, and completing training of the animation generation model.

2. The deep learning based animation generation model training method of claim 1, wherein the walking direction comprises front, rear, left and right, and the walking action comprises a left leg taking action, a right leg taking action and rest.

3. The deep learning based animation generation model training method of claim 1, wherein the inputting the initial keyframe to the first level network comprises:

and after mirror-transforming the initial key frame, inputting the initial key frame into the first cascade network.

4. The deep learning based animation generation model training method of claim 1, wherein the first level network comprises a first generation sub-network and a second generation sub-network, wherein inputting the initial keyframe to the first cascading network, and wherein determining the first cascading subsequent keyframe comprises:

inputting the initial key frame into the first generation sub-network, and determining the first generation key frame;

and inputting the initial key frame into the second generation sub-network, and determining the second generation key frame.

5. The method according to claim 4, wherein the walking direction of the initial key frame includes a first direction, the initial key frame is a first direction initial frame for representing an initial state of a person facing the first direction, the first generated key frame includes a plurality of first direction action subframes, and the walking actions of the plurality of first direction action subframes are different from each other.

6. The deep learning based animation production model training method of claim 4 wherein the second production sub-network comprises a generator, a first arbiter, wherein:

the generator comprises a plurality of convolution layers, and the generator is used for outputting and generating a prediction graph;

the first discriminator comprises a plurality of convolution layers and a full connection layer, the first discriminator is used for comparing the generated predictive graph, the initial key frame and the actual subsequent key frame corresponding to the initial key frame in the second generated sub-network, and the output of the first discriminator is a discrimination result after comparison.

7. The deep learning based animation generation model training method of claim 6 wherein the generator's loss function employs an L1 loss function and the first arbiter employs a cross entropy loss function.

8. The deep learning based animation generation model training method of claim 4, wherein the first cascading subsequent keyframes are input to the second cascading network, and wherein determining the second cascading subsequent keyframes comprises:

and inputting the second generated key frame into the second cascade network, and determining the third generated key frame.

9. The deep learning based animation generation model training method of claim 8 wherein the walking direction of the second generation key frame comprises a second direction, a third direction and a fourth direction, the second generation key frame comprises a second direction initial frame, a third direction initial frame and a fourth direction initial frame, the second cascading network comprises a third generation sub-network, a fourth generation sub-network and a fifth generation sub-network, the inputting the second generation key frame into the second cascading network, determining the third generation key frame comprises:

inputting the second direction initial frame into the third generation sub-network, and determining a plurality of second direction action sub-frames, wherein the second direction action sub-frames are identical to the walking direction of the second direction initial frame and different from each other, and the walking actions of the plurality of second direction action sub-frames are different from each other;

Inputting the third-direction initial frame into the fourth generation sub-network, and determining a plurality of third-direction action sub-frames, wherein the third-direction action sub-frames are the same as the walking direction of the third-direction initial frame and different from each other, and the walking actions of the plurality of third-direction action sub-frames are different from each other;

and inputting the fourth-direction initial frame into the fifth generation sub-network, and determining a plurality of fourth-direction action sub-frames, wherein the fourth-direction action sub-frames are the same as the walking direction of the fourth-direction initial frame and different from each other, and the walking actions of the plurality of fourth-direction action sub-frames are different from each other.

10. The deep learning based animation generation model training method of claim 9 wherein the first generation sub-network, the third generation sub-network, the fourth generation sub-network, and the fifth generation sub-network are identical, comprising a forward generator, a reverse generator, and a second arbiter, wherein:

the forward generator comprises a plurality of convolution layers, wherein the input of the forward generator is an input key frame image corresponding to the generation sub-network to which the input belongs, and the output of the forward generator is a forward generation prediction graph;

The reverse generator comprises a plurality of convolution layers, wherein the input of the reverse generator generates a prediction graph for the forward direction, and the output of the reverse generator generates the prediction graph for reduction;

the second discriminator comprises a plurality of convolution layers and a full-connection layer, the second discriminator is used for comparing the forward generation prediction graph, the input key frame image and the actual follow-up key frame corresponding to the input key frame image of the second discriminator in the generation sub-network, and the output of the second discriminator is a discrimination result after comparison.

11. The training method of the animation generation model based on the deep learning according to claim 10, wherein the loss function of the forward generator and the loss function of the reverse generator both use an L1 loss function, and the determining process of the loss function of the second discriminator comprises:

determining a first L1 loss function according to the restored prediction graph and the input key frame image;

determining a second L1 loss function according to the forward generated predictive graph and the actual subsequent key frames corresponding to the input key frame image in the sub-network to which the predictive graph belongs;

and carrying out weighted summation on the first L1 loss function and the second L1 loss function to determine the loss function of the second discriminator.

12. An animation generation model training device based on deep learning, comprising:

an acquisition unit: the method comprises the steps that a training set sequence is obtained, the training set sequence comprises a plurality of key frame sequences, each key frame sequence comprises a plurality of key frame images of different walking postures of a person, each key frame image comprises an initial key frame and a corresponding actual subsequent key frame, the initial key frame is the walking posture of the person in an initial state, and the actual subsequent key frame is the walking posture of the person after the initial state; the walking gesture comprises a walking direction and a walking action of the character;

and a processing unit: the initial key frame is input into the animation generation model, and a predicted subsequent key frame is determined; and determining a value of a loss function from the predicted subsequent key frame and the actual subsequent key frame; the animation generation model comprises a first cascade network and a second cascade network, and the predicted subsequent key frames comprise a first cascade subsequent key frame and a second cascade subsequent key frame; inputting the initial key frame into the first cascade network, and determining the first cascade subsequent key frame; inputting the first cascade subsequent key frame into the second cascade network, and determining the second cascade subsequent key frame; the first cascade subsequent key frame comprises a first generated key frame and a second generated key frame; the second cascade subsequent key frame is a third generated key frame; the first generated key frame is the same as the walking direction of the initial key frame and the walking action is different; the second generated key frame is different from the initial key frame in the walking direction and the walking action is the same; the walking direction of the third generated key frame is the same as that of the second generated key frame, and the walking actions are different;

Training unit: and the parameters of the animation generation model are adjusted according to the value of the loss function until convergence conditions are met, so that training of the animation generation model is completed.

13. An animation generation method based on deep learning, which is characterized by comprising the following steps:

acquiring an initial state key frame;

inputting the initial state key frame into an animation generation model, and determining a predicted subsequent key frame, wherein the animation generation model is obtained by training by adopting the animation generation model training method based on the deep learning according to any one of claims 1-11;

performing foreground and background segmentation on the initial state key frame and the predicted subsequent key frame, and determining a segmentation result picture;

and circularly playing the segmentation result pictures to generate animation.

14. The deep learning based animation generation method of claim 13, wherein determining a segmentation result picture comprises:

determining a transparency channel according to the result of foreground and background segmentation;

and according to the transparency channel, storing the initial state key frame and the predicted subsequent key frame as the segmentation result picture.

15. The deep learning based animation generation method of claim 14 wherein the determining a transparency channel based on the result of the foreground-background segmentation comprises:

For the foreground portion, the clear channel value is 1; for the background portion, the clear channel value is 0.

16. The deep learning based animation generation method of claim 15, wherein the saving the initial state key frame and the predicted subsequent key frame as a segmentation result picture according to the transparency channel comprises:

in the initial state key frame and the predicted subsequent key frame, the color of the part with the transparent channel value of 1 is colorless, and the color of the part with the transparent channel value of 0 is unchanged;

and saving the segmentation result picture in a png format.

17. The deep learning based animation generation method of claim 13, wherein the looping playing the segmentation result picture to generate an animation comprises looping a single play, the single play comprising:

playing the segmentation result picture corresponding to the initial state key frame;

and sequentially playing the segmentation result pictures corresponding to the predicted subsequent key frames according to the generation sequence of the predicted subsequent key frames.

18. An animation generation device based on deep learning, comprising:

an acquisition unit: the method comprises the steps of acquiring an initial state key frame;

And a processing unit: the method comprises the steps of inputting the initial state key frame into an animation generation model, and determining a predicted subsequent key frame, wherein the animation generation model is obtained by training by using the animation generation model training method based on deep learning according to any one of claims 1-11; the method is also used for carrying out foreground and background segmentation on the initial state key frame and the predicted subsequent key frame, and determining a segmentation result picture;

a playing unit: for cyclically playing the segmentation result pictures to generate animations.

19. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the deep learning based animation generation model training method according to any one of claims 1-11 or implements the deep learning based animation generation method according to any one of claims 13-17.

Images