JP5458245B2 - Operation control apparatus, method thereof, and program - Google Patents

Description

  The present invention relates to an operation control device that controls the movement of a controlled object based on user operation information.

  Currently, in interactive applications such as computer games, the degree of freedom of actions that a user can perform on a character is very limited. Current interfaces typically use input devices such as gamepads, mice, and keyboards, and perform predetermined actions such as walking, punching, and kicking when a user presses a button. is there. The movement of the character is realized by reproducing short movement data created in advance, and the user cannot freely change the style such as the posture and speed of the movement or create a new movement.

  However, an interface that allows the character to freely operate, for example, allows a character to perform various fighting actions in a fighting game, or allows avatars to perform gestures such as gestures in communication using an avatar in a three-dimensional virtual world. It is very important for applications that require interactive expression of various movements, such as creating animations in which characters dance to music.

  In recent years, a Style-based IK (Style-Based Inverse Kinematics) method has been proposed that uses a statistical approach to generate natural postures and movements that satisfy given constraints (non-kinematic calculations) (non-kinematics). (See Patent Documents 1 to 3). In Style-based IK, a learning model is constructed by mapping a large number of postures to a low-dimensional latent space in advance, and when constraints are given, the postures (coordinates) that satisfy the constraints in the latent space are searched. Calculate the pose by blending multiple surrounding poses in the latent space.

  By applying the Style-based IK with the position of each part of the character operated by the user as a constraint condition, the posture of the character can be generated. However, there are some problems in using the Style-based IK for manipulating characters. First, the first problem is that different types of movement postures cannot be expressed by a single learning model. When there are a large number of postures, there is a problem that mapping to the latent space cannot be performed well, and what posture is expected even under the same constraint condition differs depending on the operation, so that an appropriate posture cannot be obtained. As a second problem, when a trajectory of a part in a three-dimensional space is given, the trajectory in the latent space that satisfies the constraint condition of the given trajectory is not always continuous. For this reason, even if the constrained position of the part is continuously changed, the output posture is not always changed smoothly, and the operation becomes unnatural.

  In addition, as a technique for generating a character motion in response to a user operation, for example, a method for generating a motion based on a single point coordinate or trajectory input using an interface such as a mouse or a pen has been studied. ing. Many systems have been developed in which a user draws a trajectory on the ground in a virtual space using a mouse or a pen and generates a character's walking motion according to the trajectory (see Non-Patent Document 4).

When a user draws a character's waist trajectory, a system that performs actions such as jumping and somersault in addition to walking and running by selecting an action according to the shape of the input line (gesture) It has been proposed (see Non-Patent Document 5). In addition, there is a technique in which a user draws a line on the screen from an object that is a starting point of an action to an object that is an end point of the action, and selects and executes an appropriate action according to the two objects. It has been proposed (see Non-Patent Document 6).

  Furthermore, a technique has been proposed that realizes motion generation according to the trajectory of an input part by expressing the entire motion, not the posture, with a statistical model (see Non-Patent Document 7).

  Furthermore, by placing a marker representing the key posture in the virtual space in advance by the user and blending each key posture according to the distance between the mouse coordinates and each marker when the user operates the mouse. An interface that allows a user to generate an action has been proposed (see Non-Patent Document 8).

  Furthermore, a method has been proposed in which the movement of only a part of a simple articulated body is operated by a mouse operation, and the whole body movement is generated using PD control and dynamics simulation (see Non-Patent Document 3). reference).

  Furthermore, a system has been proposed in which the movement of a part of a character is input by moving a marker in a three-dimensional space (see Non-Patent Document 9).

Keith Grochow, Steven L. Martin, Aaron Hertzmann, Zoran Popovic. "Style-based Inverse Kinematics". ACM Transactions on Graphics, Vol.23, Issue 3, pp, 522-531, 2004. Hyun Joon Shin and Jehee Lee.Motion Synthesis and Editing in Low-Dimensional Spaces.Computer Animation and Virtual Worlds (CASA 2006), Volume 17, Issue 3-4, pp.219-227, 2006. Joe Laszlo, Michael Neff, Karan Singh, "Predictive Feedback for Interactive Control of Physics-based Characters", EUROGRAPHICS 2005, pp.257-265, 2005. Sang Il Park, Hyun Joon Shin, Sung Yong Shin, "On-line locomotion generation based on motion blending", Proc. Of ACM SIGGRAPH Symposium on Computer Animation 2002, pp.105-111, 2002. M. Thorne, D. Burke. M. van De Panne. "Motion Doodles: An Interface for Sketching Character Motion", ACM Transactions of Graphics (Proc. Of SIGGRAPH 2004), Vol.23, Issue 3, pp.424-431 , 2004. Masaki Oshita, "Motion Control with Strokes", Computer Animation and Virtual Worlds, Vol.16, Issues 3-4, pp.237-244, October 2005. Jianyuan Min, Yen-Lin Chen, Jinxiang Chai. "Interactive Generation of Human Animation with Deformable Motion Models", ACM Transactions on Graphics, Vol.29, Issue 1, Article No.9, 2009. Takeo Igarashi, Tomer Moscovich, John F. Hughes. "Spatial Keyframing for Performance-driven Animation", ACM SIGGRAPH / Eurographics Symposium on Computer Animation 2005, pp.253-258, 2005. Mira Dontcheva, Gray Yngve, Zoran Popovic, "Layered Acting for Character Animation", ACM Transactions of Graphics (Proc. Of SIGGRAPH 2003), Vol.22, Issue 3, pp.409-416, 2003.

  However, as described above, in order to use the style-based IK for the operation of the character, there is a problem that postures of different types of movement cannot be expressed by one learning model. In addition, when there are a large number of postures, mapping to the latent space cannot be performed well, and what posture is expected under the same constraint condition varies depending on the operation, and thus there is a problem that an appropriate posture cannot be obtained. Furthermore, even when the position of the part in the three-dimensional space is given, even if the restriction position of the part is continuously changed, the output posture does not always change smoothly, and the operation becomes unnatural. It has the problem that it ends up.

  The technique shown in Non-Patent Document 3 has a problem that it is difficult to generate a natural human movement by the proposed method. The technique shown in Non-Patent Document 4 has a problem that the actions that can be performed by the character are limited to walking and running actions. The techniques shown in Non-Patent Documents 5 and 6 have a problem that the posture and movement during the operation cannot be finely controlled because only the operation is selected and reproduced according to the input.

  The technique shown in Non-Patent Document 7 has a problem that since the entire motion is blended and reproduced, the motion cannot be stopped or returned in the middle, or the posture cannot be freely changed. The technique shown in Non-Patent Document 8 has a problem that in order to generate a natural motion, it is necessary to arrange a key posture at an appropriate position and move the pointer to an appropriate timing and position, which requires skill. Further, like the Style-based IK, there is a problem that it is difficult to cope with different types of operations with only one set of key postures. The technique shown in Non-Patent Document 9 has a problem that since only one part can be operated at a time, it is necessary to repeat an input many times in order to create an operation, which is not suitable for the operation of the operation.

  Commercially available animation software has a function that can change the key posture using IK, but cannot control the operation in real time. In principle, the movement can be controlled by combining multiple touch inputs and IK and moving multiple parts at the same time. However, in order to generate natural movement, it is necessary to operate many parts simultaneously. It is difficult to use realistically. As described above, Style-based IK has a problem that the output posture does not always change continuously even if the constraint condition is continuously changed. Not suitable for. In Non-Patent Documents 1 and 2, applications such as motion generation are performed by changing the posture of the key frame by generating the motion from the trajectory drawn in the latent space, but from the trajectory on the three-dimensional space or the screen. No action is generated.

  Therefore, in the present invention, it is noted that there are generally two types of control: posture control for changing the posture when the user operates the character, and motion control for performing some type of motion such as punching and kicking, Different learning models are used for posture control and motion control of each motion. In both controls, a posture calculation method using a learning model is properly used. The learning model to be used is switched according to the user's operation. In addition, there is a problem that a continuous posture is not always output during posture control, a problem that there is a possibility that a physically unnatural movement may occur by simply generating a posture continuously, a learning model In order to solve the problem that the posture changes suddenly at the time of switching, a motion control device and the like applying tracking control using physical simulation is provided.

The motion control device disclosed in the present application is a motion control device that controls movement of a control object based on user operation information, and an operation input unit that inputs operation information related to an operation performed by the user, and the input Based on the operation information, the constraint condition calculating means for calculating the target position of the control object and calculating the constraint condition, and posture learning model information learned based on a plurality of types of posture information given in advance And a posture control means for controlling a change in posture not having a predetermined type based on the constraint condition, and a plurality of motion learnings for each motion learned based on a plurality of types of motion information given in advance. Based on the model information and the constraint conditions, motion control means for controlling the motion of the main part of the control object having a predetermined type, and control of the posture change based on the operation information, Is based on the results calculated by the control switching means, the posture control means, and the motion control means for switching the operation of an arbitrary part of the control object, and follows the calculation result to change the attitude of the control object. Tracking control means for calculating.

  As described above, in the motion control device disclosed in the present application, based on posture learning model information and constraint conditions, a change in posture that does not have a predetermined type is controlled, and a plurality of motion learning model information and constraint conditions are set. Based on this, it is equipped with a control switching means that controls the operation of the main parts of a control object having a predetermined type and switches between these controls, so that even a large number of postures can be mapped well into the latent space. There is an effect that different types of operations can be controlled simultaneously.

  In the motion control device disclosed in the present application, the operation input unit is a multi-input that inputs operation information of a plurality of systems simultaneously.

  As described above, the motion control device disclosed in the present application is a multi-input that inputs operation information of a plurality of systems at the same time. Therefore, by controlling a plurality of types of motions and postures at the same time, the degree of freedom of movement of the control target is controlled. As a result, it is possible to control the operation reflecting the user's intention.

  In the motion control device disclosed in the present application, the follow-up control unit calculates a motion by physical simulation for a part of the control target that is not included in the operation information.

  As described above, in the motion control device disclosed in the present application, for the part of the control target that is not included in the operation information, a motion is calculated by physical simulation. Even when the movement is realized and the learning model is switched for each movement, there is an effect that a natural movement can be realized without a sudden movement change.

  In the motion control device disclosed in the present application, the control switching unit is configured to determine the position on the screen of the average of the trajectory of the arbitrary part used for learning the motion learning model information and the trajectory of the operation information, and the time. Switching is determined based on the degree of coincidence of changes.

  As described above, in the motion control device disclosed in the present application, the degree of coincidence of the change in the position and time on the screen between the trajectory average of any part used for learning the motion learning model information and the trajectory of the operation information. Therefore, since the switching determination is performed, the posture control and the motion control can be automatically and smoothly switched to achieve a smooth and natural motion.

  The motion control device disclosed in the present application determines whether to advance or return the operation of the control object when the operation information is not input or the operation information is not changed during the operation of the control object. Time determination information related to the timing for determination is set in advance, and when the operation control unit does not input or changes the operation information, the standardization time of the posture in the latent space of the previous frame And a time direction determination means for determining whether to advance the operation in the forward time direction or the operation in the reverse time direction based on the preset time determination information, and the determination result of the time direction determination means Based on the above, control is performed to advance the operation in the forward time direction or the reverse time direction.

As described above, in the operation control device disclosed in the present application, when the operation information is not input or no change occurs during the operation, the control object advances the operation in the forward time direction or the reverse time direction. Therefore, it is possible to realize a physically natural operation and an operation suitable for the situation.

  The motion control device disclosed in the present application is a stationary determination related to a timing for determining whether or not the operation of the control target is to be stopped when the operation information is not changed during the operation of the control target. Information is set in advance, and when the operation control means no longer changes in the operation information, based on the standardization time of the posture in the latent space of the previous frame and the preset stationary determination information, A stationary determination unit that determines whether or not the operation of the control target object is stationary; and based on a determination result of the stationary determination unit, the control is performed by moving the operation in the forward time direction or the reverse time direction, or before The control is performed to maintain the same operation posture as the frame.

  As described above, in the motion control device disclosed in the present application, when there is no change in the operation information during the motion, in order to determine whether the controlled object stops the motion, a physically natural motion is performed. This achieves an effect that it can realize an operation suitable for the situation.

  In the motion control device disclosed in the present application, the motion control means generates a motion posture that satisfies a constraint condition for the input operation information when the input of the operation information has changed from a previous frame, Time control determining means for determining whether the generated standardized time of the motion posture is within a predetermined threshold range from the motion posture time in the previous frame, based on the determination result of the time control determination means When the standardized time of the generated motion posture is within a predetermined threshold range, control is performed with the generated motion posture, and when it is out of the predetermined threshold range, the generated motion posture is generated. Advancing time in the latent space toward the movement posture in a predetermined time, forward time direction or backward time direction, and combining and controlling the movement posture based on the position in the latent space at the advanced time A.

  As described above, in the motion control device disclosed in the present application, by combining the motion posture as necessary, it is possible to realize a natural motion considering continuity even when the motion posture is changed suddenly. There is an effect that can be.

It is a figure which shows an example of the operation input of the operation | movement control apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the latent space for attitude | position control in the motion control apparatus which concerns on 1st Embodiment. It is a figure which shows the example of the start track | orbit of several operation | movement in the operation | movement control apparatus which concerns on 1st Embodiment. It is a figure which shows the operation state in the operation control apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the latent space for operation control in the operation control apparatus which concerns on 1st Embodiment. It is a hardware block diagram of the operation | movement control apparatus which concerns on 1st Embodiment. It is a functional block diagram of the operation control device concerning a 1st embodiment. It is a functional block diagram of the operation control part in the operation control apparatus concerning a 1st embodiment. It is a functional block diagram of the follow-up control part in the operation control device concerning a 1st embodiment. It is a flowchart which shows operation | movement of the operation control apparatus which concerns on 1st Embodiment. It is a flowchart which shows the operation control process in the operation control apparatus which concerns on 1st Embodiment. It is a flowchart which shows the follow-up control process in the motion control apparatus which concerns on 1st Embodiment.

  Embodiments of the present invention will be described below. The present invention can be implemented in many different forms. Therefore, the present invention should not be construed based only on the description of the present embodiment. Also, the same reference numerals are given to the same elements throughout the present embodiment.

(First embodiment of the present invention)
The operation control apparatus according to the present embodiment will be described with reference to FIGS. FIG. 1 is a diagram illustrating an example of an operation input of the motion control device according to the present embodiment, FIG. 2 is a diagram illustrating an example of a latent space for posture control in the motion control device according to the present embodiment, and FIG. FIG. 4 is a diagram illustrating an example of a start trajectory of a plurality of motions in the motion control device according to the present embodiment, FIG. 4 is a diagram illustrating motion states in the motion control device according to the present embodiment, and FIG. 5 is a motion according to the present embodiment. FIG. 6 is a diagram illustrating an example of a potential space for operation control in the control device, FIG. 6 is a hardware configuration diagram of the operation control device according to the present embodiment, and FIG. 7 is a functional block diagram of the operation control device according to the present embodiment. FIG. 8 is a functional block diagram of the operation control unit in the operation control apparatus according to the present embodiment, FIG. 9 is a functional block diagram of the tracking control unit in the operation control apparatus according to the present embodiment, and FIG. The operation of the motion control device Flowchart, Fig. 11 is a flowchart showing the operation control process in the operation control device according to the present embodiment, FIG. 12 is a flow chart showing a following control process in the operation control device according to the present embodiment.

  As shown in FIG. 1, the motion control apparatus according to the present embodiment controls and outputs the character posture of each frame based on multi-touch input. The input device may be any device that supports multi-input, and may be configured to use a plurality of pointers, a plurality of mice, and the like.

  When a user touches and drags a character part by multi-touch input, a part / target position constraint condition is generated for each touch point, and an output posture is calculated based on the constraint condition. Here, since the constraint condition of the target position is a position on the two-dimensional screen, it is actually a straight line in the three-dimensional space.

  As described above, in the present embodiment, the learning model is switched and used depending on the posture control and the motion control. For motion control, a different learning model is used for each motion type. The system uses attitude control in the initial state. If the trajectory when the user moves the part of the character is close to any of the movements prepared in advance, the movement control is performed by switching to the learning model of the movement. When the operation being executed is terminated by the user's input, the state returns to the state in which the posture control is used.

  In this embodiment, in order to realize a natural posture change, the posture by the posture control / motion control is not output as it is, but the output posture is generated by using the follow-up control. In this embodiment, the action when touching a place other than the character's part and the action when performing a multi-touch gesture such as pinch / swipe are not particularly defined. , For example, can be assigned to another function such as a viewpoint operation.

  Posture control will be described. The posture control in the present embodiment is intended for a motion operation that does not fit in a specific motion type, such as a character changing posture or moving a hand or a foot freely. In the present embodiment, the attitude is generated using the Style-based IK technique. As the data structure, different attitude calculation methods are used for attitude control and motion control.

  In posture control, a model learned based on posture information given in advance is used. There are several methods for mapping the sample posture to a low-dimensional latent space (in this embodiment, a two-dimensional latent space is used) (for example, see Non-Patent Documents 1 and 2). In the implementation of, a method is used in which the distance between sample postures is mapped as close as possible in the latent space by using a multidimensional scaling method (MDS). The distance between sample postures is calculated by averaging the distances of all joints after moving and rotating both postures so that the waist position and body orientation match.

  As the sample posture used for learning, the posture is extracted from the motion data including various posture changes when the character is standing. In addition, when the character performs each motion, the initial posture of the motion before the motion start determination is generated using the posture control, so the motion data used for learning each motion is started. The position of the part is also used (details of operation start determination and operation control will be described later). For all sample postures, the mirrored postures are used for learning. FIG. 2 shows an example of a latent space of learning results.

  When a target position of one or a plurality of parts is designated by an input from the user, a posture close to the designated constraint condition is generated using a learning model. The coordinates of the latent space corresponding to the specified constraint condition are searched, and the attitude is generated by synthesizing surrounding sample attitudes.

  It is possible to efficiently search by dividing the latent space into grids, obtaining representative sample postures for each grid, and comparing the input constraint conditions with them. At this time, the constraint condition that the user inputs on the screen is a two-dimensional coordinate on the screen, and therefore corresponds to a straight line in the three-dimensional space. By calculating the average value of the distance between this straight line and the position of the part of the sample posture, the distance between the restricted posture and the sample posture can be obtained. When a grid that best matches the input constraint is found, the sample posture of the surrounding grid and the distance between the constraints are calculated, and the posture is calculated by combining (weighted average) the sample postures according to the distance. Posture synthesis is performed by blending joint rotations. The position / orientation of the waist is calculated by blending the relative movement / rotation from the starting state of the original motion when the sample posture is taken out and adding it to the character's position / orientation at the start of posture control.

  Since the posture generated by Style-based IK is a sample posture that is combined so as to satisfy the constraint as much as possible, the constraint is not completely satisfied. Therefore, in the present embodiment, the conventional IK calculation is applied to the posture of the output result so that the constraint condition is satisfied. In the implementation of this embodiment, a relatively high speed CCD (Cyclic-Coordinate-Decent) IK method is used.

  As described above, in Style-based IK, when a continuous trajectory of a constraint condition is given in a three-dimensional space, it is not guaranteed that the trajectory will be a continuous trajectory in the latent space. Even if it is moved, the coordinates in the latent space move to discontinuous points, and as a result, the posture may change suddenly. In the present embodiment, this problem is not taken into account in the posture control, and the posture is simply generated, and the output posture is made continuous by adding the tracking control process. As with the motion control described later, a method of gradually changing the coordinates in the latent space can be used, but in posture control where there is no particular type of motion, such a method is used for continuous posture change. Even if generated, it is not always a meaningful operation, so the above method is used.

In addition, when generating an action by Style-based IK, it is not always natural as an action because it simply generates postures that meet the constraints. For example, if it is a real human movement, if you move a part suddenly, you cannot stop the movement suddenly, so the body moves even after stopping the movement, and the force is also applied to parts other than the part that moved Movements such as shaking are transmitted. In order to reproduce such an operation, it is necessary to generate the operation in consideration of dynamic elements by some method. In the present embodiment, this problem is also solved by using the follow-up control described later.

  Next, operation control will be described. Motion control is intended for the operation of some type of motion. If the user inputs any action to the character during the posture control, the control is switched to the action control corresponding to the action.

  The motion start condition is determined by the distance between the trajectory on the screen where the user moves the part and the trajectory projected on the motion screen of the part of the motion. For each action, manually specify the main part of the action manually (for example, if the action is punching with the right hand, the main part is the right hand), and the average trajectory of that part of all action data used for learning that action Ask for. FIG. 3 shows an example of an operation start trajectory.

  When the user moves the part, the start of motion control is determined from the degree of coincidence of the position and time changes on the screen between the input trajectory and the motion trajectory. In calculating the degree of coincidence, both trajectories are represented by an arrangement of screen coordinates and time, and first, the combination of the closest points is obtained in order from the top by dynamic programming. Then, the degree of coincidence is obtained by the sum of the average value of the distances between the points on the screen and the average value of the relative time difference from the input / operation start time. If the degree of coincidence is less than or equal to the threshold, the operation is switched to the operation control.

  When determining whether to start an operation, depending on the viewpoint, the operation trajectory becomes nearly perpendicular to the screen, making it difficult to input the trajectory or increasing the possibility of erroneous recognition. Therefore, a vector representing the entire trajectory is calculated in advance, and an operation in which the angle between the vector and the line-of-sight vector is equal to or smaller than a certain value is excluded from the start determination candidates. For example, when the user wants the character to perform a punching action, the character cannot be viewed from the front, but the punching action is performed after the viewpoint is operated and the character is viewed from the side. It is necessary to move the hand of the character. In general, when a user wants a character to perform some action, it is natural to change the viewpoint so that the user can easily input a movement. Therefore, it is considered appropriate to introduce such restrictions on the viewpoint. In the example of FIG. 3, a trajectory of an executable action is indicated by a dotted line, and a trajectory of an impossible action is indicated by a solid line.

  In motion control, motion is basically generated by the same method as that used in posture control. However, in operation control, it is necessary to operate according to the type. For example, in a punching operation, even if the user suddenly moves the position of the hand forward, the posture does not suddenly match the position but needs to be in the posture while performing the punching operation. . However, in motion control, not only the motion is always executed in the forward direction, but the motion may be returned in the reverse direction or the same motion may be repeated, so it is necessary to cope with the generation of such motion. is there.

  On the other hand, even if the user continues to input so as to fix the position of the part, it is not desirable to stop the movement in an unnatural posture. For example, if you stand still in the middle of moving your feet while walking, or if you stop in the air while jumping, it will result in an unnatural motion that is not physically correct. Similarly, when the user stops input during the operation (releases all fingers), the user returns to the posture at the start of the operation, or the operation is executed to the end, instead of being stopped in the posture during the operation. It is expected to do. In the present embodiment, the posture is generated in consideration of such restrictions.

First, in the learning process of each motion, when mapping the sample posture to the latent space, not only the posture but also the motion time of the sample posture (the motion start time with the original motion data is set to 0, and the end time is set to 1). Time standardization time) and the following operating states are recorded.
(1) Standstill propriety: Whether to allow or not to stand still (2) Automatic playback: Whether to play backward to the start posture or to play forward to the end posture when the operation is stopped Since is independent, the state is set for each type. For example, a state section may be manually specified for each motion data, and the state of each sample posture may be determined based on the information (see FIG. 4). An example of a latent space for operation control is shown in FIG. In FIG. 5, continuous postures (points) in the original motion are connected by a line, and each point also includes time information.

  At the time of posture generation, motion generation considering continuity is performed in order to satisfy the above purpose. First, as in the attitude control, the coordinates in the latent space are searched based on the input constraint conditions. At this time, not only the posture corresponding to the coordinates but also the standardized time of the operation is obtained by blending. When the motion time of the previous frame is far from the motion time of the search posture, instead of outputting the search posture, the posture advanced (or returned) by a certain time in the latent space toward the search posture is output. To do. In each sample posture, a vector to the posture at the next time is recorded to determine which direction to proceed in the latent space.

  When the user stops input during motion control, it decides whether to play in the forward direction or the backward direction based on the state of the current posture. Find a vector and generate a pose. When the movement end posture is reached, the motion control is ended and the posture control state is restored. When the control method is switched between posture control and motion control, even if both include a common sample posture, the posture may change abruptly, resulting in an unnatural motion. In this embodiment, this problem is also solved by introducing tracking control described later.

  Next, the follow-up control will be described. The purpose of the follow-up control is to generate a natural motion by changing the character's posture based on the target posture calculated by the posture control / motion control. As described above, posture control has problems that posture does not always change continuously, and that movement based on physical laws such as shaking when the body is moved cannot be handled. Also, the posture may change suddenly when switching between posture control and motion control. In the present embodiment, these problems are solved by applying tracking control using physical simulation.

  In the follow-up control, PD control that follows the target posture is performed using the posture output by the posture control or motion control as the target posture, and the posture change of the character is calculated by physical simulation. Non-patent Document 3 and Reference Document 1 (Victor B. Zordan, Anna Majkowska, Bill Chiu, Matthew Fast. “Dynamic Response for Motion Capture Animation”, ACM Transaction of Graphics (Proc. Of SIGGRAPH 2005), Vol. 24, Issue 3, pp.697-701, 2005), etc., but because individual joints are controlled independently so as to approach the target posture, it becomes a human-like natural movement such as slow movement. Cannot be reproduced. Therefore, in the present embodiment, the posture of the follow-up control result is used only for a portion that is not directly operated by the user during posture control and immediately after control switching. For other parts, postures based on posture control / motion control are used as they are.

In the present embodiment, a posture by tracking control is calculated using the same method as in Reference Document 1. When the target posture is determined from the output of the posture control / motion control, the torque of each joint is calculated by PD control. The posture change is calculated by performing a dynamic simulation according to this torque. Regarding the posture change of the lower body, similarly, PD control is used to determine the translational movement force / rotation torque of the waist and apply it to the dynamic simulation. At this time, the result of the follow-up control is used only for the joint of the part of the whole body posture that is not operated by the user. The human body is divided into the range of right shoulder to right hand, left shoulder to left hand, waist to neck, and lower body, and if there is no part that the user is operating for each range, the joint angle resulting from the follow-up control Is used. By using tracking control, continuous posture change is realized. For example, when the right arm is moved during posture control, movements that are not included in the learning model can be reproduced, such as the hips and left arm moving slightly due to shaking.

  FIG. 6 shows a hardware configuration of the operation control apparatus according to the present embodiment. The motion control device 1 stores a ROM 13 in which system programs and various programs (for example, posture control programs and motion control programs) are stored, and various data (for example, posture learning information and motion learning information). An HD (Hard Disk) 14, various programs and the like are read out as necessary, a RAM 12 used as a work area of the CPU 11, a CPU 11 that executes an actual calculation based on the read programs and the like, and other devices Accepts input from a communication I / F 15 that is an interface for communicating with (for example, another terminal that is online through a network), and input devices such as a controller, a game pad, a keyboard, and a mouse, and a display , An interface for outputting data to a printer, etc. That and an input and output I / F16. The input / output I / F 16 includes, for example, USB, RS232C, etc. in addition to optical discs such as CD-ROM and DVD-ROM, drives corresponding to game cassettes, etc., which store game programs in home game devices. .

  FIG. 7 shows a functional block diagram of the operation control apparatus according to the present embodiment. The motion control apparatus 1 includes an operation information input unit 21, a constraint condition calculation unit 22, a motion selection / switching unit 23, a motion control unit 24, a posture control unit 25, a follow-up control unit 26, posture learning model information 27, and motion learning model information. 28 and a display 30. The operation selection / switching unit 23 further includes an operation selection unit 231 and a switching unit 232.

  The operation information input unit 21 inputs operation information 29 related to the operation performed by the user. The operations performed by the user include operations using a controller, a game pad, a keyboard, a mouse, etc. In order to increase the freedom of movement of characters on the display, a multi-touch device is particularly used in this embodiment. Is desirable.

  Based on the operation information 21 (here, multi-touch operation information using a multi-touch device) input by the operation information input unit 21, the constraint condition calculation unit 22 is set for each touch point touched by the user. A constraint condition for the part / target position (coordinates on the two-dimensional screen) is generated.

  The motion selection / switching unit 23 receives an operation of a certain type during a posture control that does not fit in a specific motion type in which the character changes the posture or moves the limbs freely. The action selection unit 231 selects the action learning model information 28 corresponding to the action, and the switching part 232 switches to the action control. Specifically, as described above, on the screen between the average trajectory of the relevant part of all the motion data used for learning the motion and the input trajectory input by the operation, specifying the main part of the motion Switching is determined based on the degree of coincidence of the position and time changes.

  The posture control unit 25 calculates an output posture based on the constraint condition and the posture learning model information 27 learned based on the posture information given in advance, so that a motion that does not fit in a specific motion type can be obtained. Control.

The motion control unit 24 calculates the output posture based on the motion learning model information 28 corresponding to the selected motion when the motion selection / switching unit 23 switches from the posture control to the motion control and selects a specific motion. By doing so, you control the typed action. Here, when the motion learning model information 28 is created, the standard time of motion at each coordinate in the latent space is recorded, and when the motion posture is generated, only the posture corresponding to the coordinate in the latent space is used. In addition, the continuity is realized as described above by obtaining the standardized time of the operation.

  Here, the operation control unit 24 will be described in more detail. In FIG. 8, the motion control unit 24 includes a stillness determination unit 31, a time direction determination unit 32, a motion time control determination unit 33, a motion time control unit 34, and an attitude calculation unit 35.

  For example, when there is an input that fixes the position of the part (multi-touch touch position is fixed), the stationary determination unit 31 sets in advance whether the operation is stationary even during the operation and the standardized time The determination is made based on the timing information (see FIG. 4) in the section of the performed operation. When stationary, the posture calculation unit 35 performs posture calculation to maintain the same motion posture as the previous frame, and when not stationary, the time direction determination described below is performed.

  For example, the time direction determination unit 32 is in a state where there is an input that fixes the position of the part but the operation cannot be stopped, or the user quits the input in the middle of the operation (from multi-touch to all fingers In the case where the time is advanced in the forward direction or in the reverse direction, it is determined based on the standardized time and the timing information (see FIG. 4) in the preset operation period. . Depending on the determination result, the operation time control unit 34 controls the time in the forward time direction or the reverse time direction, and the posture calculation unit 35 calculates the posture.

  The motion time control determination unit 33 sets the standardization time of the motion posture satisfying the constraint condition for the input operation information (the standardization time in the calculation information calculated by the posture calculation unit 35 once) as the potential of the previous frame. It is determined whether it is within a predetermined time range from the standardized time of the motion posture in the space. If it is within the range of the predetermined time, the posture calculation unit 35 calculates the motion posture as it is. If it is out of the predetermined time range, the motion time control unit 34 controls the time in the forward time direction or the reverse time direction, and the posture calculation unit 35 synthesizes the motion posture at that time to calculate the posture. Do.

  Returning to FIG. 7, the tracking control unit 26 generates a natural motion by changing the posture based on the control of the posture control unit 25 and the target posture calculated by the motion control unit 24, and outputs it to the display 30. To do.

  Here, the follow-up control unit 26 will be described in more detail. In FIG. 9, the follow-up control unit 26 includes a joint torque calculation unit 41, a physical simulation unit 42, an input part specifying unit 43, and an output posture specifying unit 44. This follow-up control unit 26 does not perform processing during motion control, and uses the motion posture as it is as the output posture.

  The joint torque calculator 41 calculates joint torque by PD control in order to bring the posture of the previous frame closer to the target posture. Based on the calculated joint torque, the physical simulation unit 42 performs a dynamic simulation to obtain a posture. The posture obtained here is the updated posture. The input part specifying unit 43 specifies a part where the operation information is input and a part where the operation information is not input. The output posture setting unit 44 sets the joint angle of the target posture as the output posture for a portion where the operation information is input, and sets the joint angle of the updated posture as the output posture for a portion where the operation information is not input. .

FIG. 10 is a flowchart illustrating a processing procedure of the operation control apparatus. First, input information by multi-touch is input to the operation information input unit 21 as operation information 29 (S11). The constraint condition calculation unit 22 calculates a constraint condition between the part and the target position for each touch point (S12).
In the initial state, the posture control unit 25 performs posture control (S13), and when an operation is input, the operation selection / switching unit 23 selects the corresponding operation and switches between the controls. The motion control unit 24 performs motion control based on the selected motion learning model information (S14).

  Here, the processing procedure of the operation control in step S14 will be described in more detail with reference to FIG. When the operation control is started, it is first determined whether or not operation information has been input (S21). If there is no input, the time direction determination unit 32 performs timing information in the standardized time and a preset operation interval. Based on the above, the time direction is determined (S25). In the case of the forward time direction, the motion posture is advanced and controlled in the forward time direction (direction to advance to the target posture) (S26), and in the case of the reverse time direction, the motion posture is reversed in the reverse time direction (direction to return to the previous posture). The process proceeds to (S27) and the process is terminated.

  If there is an input in S21, it is determined whether or not the input operation information is the same as the previous frame (S22), and if it is the same, the stillness determination unit 31 performs a stillness determination (S23) and is different. In this case, the operation time control determination unit 33 determines whether to control the operation time (S28). If it is determined in S23 that the stillness determination unit 31 is not stationary, the processing after S25 described above is performed. If it is determined that the stillness determination unit 31 is still, control is performed to maintain the same motion posture as the previous frame (S24), and the process ends.

  When the operation time control determination unit 33 determines not to control the operation time in S28, the operation posture is controlled as it is according to the constraint condition on the input operation information (S29), and the process is terminated. When it is determined that the operation time is to be controlled, the operation posture at the time advanced by a predetermined time in the forward time direction or the reverse time direction is synthesized and controlled (S30), and the process ends. The time advanced by the predetermined time referred to here is within a range less than the predetermined time used when the composition determination unit 33 determines whether to perform composition.

  Returning to FIG. 10, the target posture is calculated by the posture control and motion control (S15), and the tracking control unit 26 performs tracking control (S16).

  Here, the processing procedure of the follow-up control in step S16 will be described in more detail with reference to FIG. When follow-up control is started, it is first determined whether the control mode is motion control or attitude control (S31). In the case of motion control, the target posture is output as it is (S32, S33), and the process ends. In the case of posture control, a joint torque is calculated (S34), and a dynamic simulation based on the joint torque is executed (S35). Identify parts with and without input of operation information, set parts with inputs as motion postures to set and output joint angles as target postures, and parts with no inputs by dynamic simulation The obtained updated posture is set as a motion posture to be set and output (S36), the motion posture is output (S33), and the processing is terminated.

  Returning to FIG. 10, the output posture subjected to the follow-up control is displayed on the display 30 (S17), and the process ends.

  Although the present invention has been described with the above-described embodiments, the technical scope of the present invention is not limited to the scope described in the embodiments, and various changes or improvements can be added to these embodiments.

The technique shown in the above embodiment was implemented. Using this installed system, it was confirmed that the characters can be operated and the operations as expected can be performed (see FIG. 1). Moreover, about the determination at the time of moving from posture control to motion control, it was determined substantially correctly. In addition, depending on the position of the input part, there was rarely an unnatural posture, but this applied the conventional IK after the Style-based IK so as to satisfy the input constraint condition. This is probably because the posture is corrected. This problem can be solved by adding a larger number of sample postures or limiting posture correction by IK.

  As a result of actually operating the character using the developed system, according to the operation of the user, the action that the user is going to perform is selected, and the device has been devised to realize that action, The expected motion could be realized by moving only one part. Furthermore, when multiple operation inputs were performed simultaneously with multi-touch, it was possible to move a plurality of parts during posture control, or change the posture by changing posture during motion control. In addition, when manipulating the character, it is not always necessary to move the part, but it may be necessary to fix a part so as not to move, and such a request can also be realized by multi-touch.

  If you operate by touching the screen directly, it may be difficult to see because the screen is hidden by hand. For this reason, there is a problem that the character part cannot be touched until the previous action is completed, and the next action cannot be input in advance. As for these problems, there are conceivable solutions such as using a plurality of screens or displaying the posture at the end of the operation so that it can be operated.

  Further, as a problem related to the types of operations that can be operated, there is a problem that operations other than pre-registered operations cannot be realized. Also, it is difficult to realize a combination of a plurality of operations such as waving hands while walking. With regard to the movement of the character, it is currently possible to execute a walking action or a direction changing action in accordance with the movement of the hips, but it is difficult to walk along an arbitrary trajectory. With regard to the moving motion, an extension may be considered in which a character is walked according to the trajectory by dragging the waist more than a certain distance using a special motion determination / generation method.

1 Operation control device 11 CPU
12 RAM
13 ROM
14 HD
15 Communication I / F
16 Input / output I / F
DESCRIPTION OF SYMBOLS 21 Operation information input part 22 Constraint condition calculating part 23 Action selection / switching part 231 Action selection part 232 Switching part 24 Action control part 25 Attitude control part 26 Follow-up control part 27 Attitude learning model information 28 Action learning model information 29 Operation information 30 Display 31 stationary determination unit 32 time direction determination unit 33 synthesis determination unit 34 motion posture synthesis unit 35 control processing unit 41 joint torque calculation unit 42 physical simulation unit 43 input part specification unit 44 output posture setting unit

Claims (9)

In the motion control device that controls the movement of the controlled object based on the operation information of the user,
Operation input means for inputting operation information related to the operation performed by the user;
Based on the input operation information, a constraint condition calculation unit that calculates a constraint condition by calculating a target position of the control object;
Posture learning model information learned based on a plurality of types of posture information given in advance, and posture control means for controlling a change in posture not having a predetermined type based on the constraint condition;
A plurality of motion learning model information for each motion learned based on a plurality of types of motion information given in advance, and a motion of an arbitrary part of the control object having a predetermined type based on the constraint condition Operation control means for controlling
Control switching means for switching posture control or motion control based on the operation information;
An operation control apparatus comprising: the attitude control unit; and a tracking control unit that calculates the attitude of the object to be controlled by following the calculation result based on the result calculated by the operation control unit. The operation control apparatus according to claim 1,
The operation control device is characterized in that the operation input means is a multi-input that inputs operation information of a plurality of systems simultaneously. In the operation control device according to claim 1 or 2,
The motion control apparatus, wherein the follow-up control means calculates a motion by physical simulation for a part of the control object that is not included in the operation information. In the operation control device according to any one of claims 1 to 3,
The control switching means switches based on the position on the screen of the trajectory average of the arbitrary part used for learning the motion learning model information and the trajectory of the operation information, and the degree of coincidence of changes in time. An operation control apparatus characterized by performing the determination. In the operation control device according to any one of claims 1 to 4,
Time determination information related to timing for determining whether to advance or return the operation of the control object when the operation information is not input or the operation information is not changed during the operation of the control object. Is preset,
The operation control means is
Whether the operation is advanced in the forward time direction based on the standardized time of the posture in the latent space of the previous frame and the preset time determination information when the operation information is not input or no longer changes Or a time direction determination means for determining whether to move the operation in the reverse time direction,
An operation control apparatus that performs control to advance an operation in a forward time direction or a reverse time direction based on a determination result of the time direction determination means. In the operation control device according to any one of claims 1 to 5,
In the operation of the control object, when there is no change in the operation information, stationary determination information related to the timing for determining whether to stop the operation of the control object is preset,
The operation control means is
When there is no change in the operation information, it is determined whether or not the operation of the control object is to be stopped based on the standardization time of the posture in the latent space of the previous frame and the preset stationary determination information. A stationary determination means for
Based on the determination result of the stillness determination means, the operation is controlled by advancing the operation in the forward time direction or the reverse time direction, or performing the control to maintain the same operation posture as the previous frame. . In the operation control device according to any one of claims 1 to 6,
The operation control means is
When the input of the operation information has changed from the previous frame, an operation posture that satisfies the constraint condition for the input operation information is generated, and the standardized time of the generated operation posture is the operation in the previous frame. Time control determination means for determining whether or not the posture time is within a predetermined threshold range,
Based on the determination result of the time control determination means, when the standardized time of the generated motion posture is within a predetermined threshold range, control is performed with the generated motion posture, and the predetermined threshold range When it is outside, the time is advanced in the latent space toward the generated motion posture in a predetermined time, forward time direction or reverse time direction, and the motion is based on the position in the latent space at the advanced time. A motion control apparatus characterized by combining and controlling a posture. In an operation control method in which a computer controls the movement of an object to be controlled based on user operation information,
An operation input step of inputting operation information related to the operation performed by the user;
A constraint condition calculating step for calculating a constraint condition by calculating a target position of the control object based on the input operation information;
Based on the operation information, control switching means for switching the control of the posture change, or the control of the operation of the main part of the control object;
A posture for controlling a change in posture not having a predetermined type based on posture learning model information learned based on a plurality of types of posture information given in advance and the constraint conditions in accordance with switching of the control switching means. Based on the control processing or a plurality of motion learning model information for each motion that has been learned based on a plurality of types of motion information given in advance and the constraint condition, any of the control objects having a predetermined type A control step for executing a motion control process for controlling the motion of the part;
A motion control method comprising: a posture control step; and a follow-up control step for calculating a posture of the control object based on the calculation result based on the result calculated by the posture control step and the motion control step. In an operation control program that causes a computer to function to control the movement of an object to be controlled based on user operation information,
Operation input means for inputting operation information relating to the operation performed by the user;
Based on the input operation information, a constraint condition calculation unit that calculates a constraint condition by calculating a target position of the control object,
Posture learning model information learned based on a plurality of types of posture information given in advance, and posture control means for controlling a change in posture not having a predetermined type based on the constraint condition;
A plurality of motion learning model information for each motion learned based on a plurality of types of motion information given in advance, and the motion of the main part of the control object having a predetermined type based on the constraint condition Operation control means for controlling,
Based on the operation information, control switching means for switching control of posture change, or control of operation of an arbitrary part of the control object,
An operation control program for causing a computer to function as follow-up control means for calculating the attitude of the object to be controlled based on results calculated by the attitude control means and the action control means.