KR102899190B1 - Method and system for generating gesture-enhanced realistic digital human tutor - Google Patents

Abstract

The present disclosure relates to a method and system for generating a gesture-augmented digital human tutor (DHT). The method for generating a digital human tutor includes: acquiring a half-body or full-body image of an instructor and a lecture video; generating a digital human tutor corresponding to the instructor's facial image using the instructor's facial image; extracting external features of the instructor's face and gestures during a lecture from the lecture video; and activating the digital human tutor by reflecting the external features to the digital human tutor.

Description

Method and system for generating gesture-enhanced realistic digital human tutor

The present disclosure relates to a method for creating a digital human tutor existing in a virtual space, and more particularly, to a method and device for creating a digital human tutor with augmented gestures that empathize with a learner's responses.

As the demand for non-face-to-face services increases due to the recent spread of viral infections, the disinhibition effect, which is a lack of empathy due to the increase in indirect communication through online channels rather than face-to-face communication, is emerging as a major problem.

Therefore, to enhance empathic interaction between instructors and learners, it is necessary to recognize learners' attitudes remotely and adjust instructor responses or feedback accordingly.

When instructors mimic the learners' various responses, learners feel a sense of similarity between themselves and the instructor, which increases social bonding and tends to empathize more with the instructor.

Therefore, technology that recognizes learners' facial expressions or reactions in real time and enables a virtual instructor, a digital human tutor, to react or imitate those facial expressions can induce empathy in learners and enhance learning effectiveness.

Joinson, A. N. (2007). Disinhibition and the Internet. In Psychology and the Internet (pp. 75-92). Academic Press. Suler, J. (2004). The online disinhibition effect. Cyberpsychology and behavior, 7(3), 321-326. Chartrand, T. L., and Van Baaren, R. (2009). Human mimicry. Advances in experimental social psychology, 41, 219-274.] Schutte, N. S., and Stilinovi?, E. J. (2017). Facilitating empathy through virtual reality. Motivation and emotion, 41(6), 708-712. Choi, Wonkyung (2020). Comparison of satisfaction with face-to-face and virtual lectures. Journal of English Education, 19(4): 223-245. Kim Sang-mi (2020). Analysis of domestic media reports on online education related to COVID-19. Journal of the Korea Digital Contents Society, 21(6), 1091-1100. Yoon, Bo-ram. (2018). The effect of virtual avatar appearance on users' social presence in an augmented reality-based remote collaboration system. Heidicker, P., Langbehn, E., and Steinicke, F. (2017, March). Influence of avatar appearance on presence in social VR. In 2017 IEEE Symposium on 3D User Interfaces (3DUI) (pp. 233-234). IEEE. Zibrek, K., Kokkinara, E., and McDonnell, R. (2018). The effect of realistic appearance of virtual characters in immersive environments-does the character's personality play a role?. IEEE transactions on visualization and computer graphics, 24(4), 1681-1690. Lee, Woo-ri, and Hwang, Min-cheol (2014). Emotion recognition accuracy for standard Korean facial expression images. Journal of the Korea Contents Association, 14(9), 476-483. Jo, D., Kim, K. H., and Kim, G. J. (2017). Effects of avatar and background types on users’ co-presence and trust for mixed reality-based teleconference systems. In Proceedings the 30th Conference on Computer Animation and Social Agents (pp. 27-36).

The present disclosure proposes a method and system for creating a gesture-augmented digital human tutor that can enhance the presence of a digital human tutor in a virtual space and effectively induce empathy from a learner.

The present disclosure proposes a method and device for generating a gesture-augmented digital human tutor that realistically creates a digital human tutor by imbuing the digital human tutor, which is a virtual avatar, with the external features including the online instructor's gestures and the unconscious micro-expressions of the instructor's face.

The present disclosure presents a method and system for creating a gesture-augmented digital human tutor that can effectively induce learner responses by augmenting an instructor's gestures to recognize and imitate learner responses in a non-face-to-face learning situation.

A method for creating a digital human tutor according to the present disclosure:

A step of acquiring a real professor's face image and lecture video by a camera;

A step of extracting the instructor's joint (keypoint) coordinates and facial features from the above real-life image by an image processor;

A step of determining an augmentation weight for the change in the external feature and the value of the joint coordinate from the external feature of the face by a characteristic processor;

A step of generating a digital human tutor corresponding to the instructor's facial image using the instructor's facial feature image by the character generation unit; and

A step of activating a digital human tutor that reflects the external characteristics to the digital human tutor by the character control unit and changes the joint coordinates of the digital human tutor by the augmented joint coordinates; is included.

According to a specific embodiment of the present disclosure, the augmented joint coordinates may correspond to the upper body area of an actual instructor.

According to a specific embodiment of the present disclosure, the joint coordinates are associated with a specific part of the human body including the joint.

According to a specific embodiment of the present disclosure,

Step of extracting the above joint coordinates:

A step of extracting joint coordinates on a two-dimensional plane by the image analysis unit; and

It may include a step of extracting three-dimensional joint coordinates (x, y, z) by inferring a third direction (z) perpendicular to the two-dimensional plane using a 3D analyzer.

According to another embodiment of the present disclosure,

In the step of determining the above augmentation weight,

In the above 3D joint coordinates, a weight can be determined for at least one joint coordinate among the three coordinates x, y, and z.

According to another embodiment of the present disclosure,

The above digital human is a digital human tutor (DHT) that leads learning through video, and a learning video can be displayed together with the DHT on the display.

According to one or more embodiments, feature points can be extracted from the facial image, and the appearance of the digital human tutor can be set using the feature points.

According to one or more embodiments, the feature points may be selected from landmarks defined in FACS.

According to one or more embodiments, it is possible to extract feature points of an instructor from a lecture video, extract motion data of the extracted feature points, and extract micro-expression data from the motion data.

According to one or more embodiments, the micro expression data (MED) may be calculated by applying a KLT (Kanade-Lucas-Tomasi) tracking algorithm or a TM (Transformation Matrix) based tracking algorithm to the feature point tracking to extract the micro expression data.

According to one or more embodiments, in order to extract unconscious micro-expression data from the micro-expression data, filtering of a predetermined frequency is performed on the micro-expression data, and the periodicity of the heartbeat is determined by principal component analysis (PCA) on the filtered micro-expression data, and the periodicity can be used as an input value for the micro-expression of the digital human tutor.

According to one or more embodiments, the external features of the instructor are extracted as landmarks defined in FACS, and the external features can be reflected in the digital human tutor in units of AUs based on the landmarks.

A system for creating a digital human tutor according to the present disclosure:

One or more cameras that capture actual instructor facial images and lecture videos;

An image processor that extracts joint (keypoint) coordinates and facial features of an actual person from the above-mentioned real-life image;

A feature processor that extracts changes in facial appearance features and joint coordinates from facial images during the lecture of the above professor, and determines augmentation weights for the values of the joint coordinates;

A character generation unit that generates a digital human tutor corresponding to the instructor's facial image using the instructor's facial image; a character control unit that reflects the external features to the digital human tutor and activates the digital human tutor to change the joint coordinates of the digital human tutor by the augmented joint coordinates; and

It includes a lecture video generation unit that generates a lecture video including the above digital human tutor.

According to a specific embodiment of the present disclosure, the augmented joint coordinates may correspond to the upper body area of an actual instructor.

According to a specific embodiment of the present disclosure, the joint coordinates are associated with a specific part of the human body including the joint.

According to a specific embodiment of the present disclosure,

Step of extracting the above joint coordinates:

A step of extracting joint coordinates on a two-dimensional plane by the image analysis unit; and

It may include a step of extracting three-dimensional joint coordinates (x, y, z) by inferring a third direction (z) perpendicular to the two-dimensional plane using a 3D analyzer.

According to another embodiment of the present disclosure,

In the step of determining the above augmentation weight,

In the above 3D joint coordinates, a weight can be determined for at least one joint coordinate among the three coordinates x, y, and z.

According to another embodiment of the present disclosure,

The above digital human is a digital human tutor (DHT) that leads learning through video, and a learning video can be displayed together with the DHT on the display.

According to one or more embodiments, the character generation unit can extract feature points from the facial image and set the appearance of the digital human tutor using the feature points.

According to one or more embodiments, the character generator can select from landmarks defined in FACS as the above-described feature points.

According to one or more embodiments, the feature processor may extract feature points of the instructor from a lecture video, extract motion data of the extracted feature points, and extract micro-expression data from the motion data.

According to one or more embodiments, the feature processor may apply a KLT (Kanade-Lucas-Tomasi) tracking algorithm or a TM (Transformation Matrix) based tracking algorithm to the feature point tracking to extract the micro-expression data.

According to one or more embodiments, the characteristic processor performs filtering of a predetermined frequency on the micro-expression data to extract unconscious micro-expression data from the micro-expression data, determines the periodicity of the heartbeat by principal component analysis (PCA) on the filtered micro-expression data, and uses the periodicity as an input value for the micro-expression of the digital human tutor.

According to one or more embodiments, the feature processor extracts the external features of the instructor as landmarks defined in FACS, and can reflect the external features of the digital human tutor in units of AUs based on the landmarks.

Figure 1 is a flowchart showing a process for detecting facial and joint features of an instructor according to one or more embodiments.
Figure 2 illustrates an arrangement of facial landmarks defined in the Facial Action Coding System (FACS).
FIG. 3 shows the results of a facial video-face detection-facial landmark detection process according to one or more embodiments.
FIG. 4 is a flowchart showing a process for extracting unconscious facial micro-expressions caused by heartbeats from a facial region in an instructor image captured by a camera, according to one or more embodiments.
Figure 5 is a flowchart of a process for determining a heartbeat signal, including a sliding window technique for the aforementioned micro-expression data (MED).
FIG. 6 is a flowchart illustrating a concept of a gesture augmentation method according to one or more embodiments.
FIG. 7 illustrates several joints extracted from a real person in a gesture augmentation method according to one or more embodiments.
FIG. 8 shows an original image of a digital human tutor (DHT) without any gesture augmentation method applied according to one or more embodiments.
FIGS. 9 to 12 are images showing the results of augmentation of the hand portion of DHT according to x, y, and z coordinates in a gesture augmentation method according to one or more embodiments.
FIG. 12 is an image showing the result of a gesture augmentation method according to one or more embodiments in which the coordinates of the hand are augmented in the x, y, and z directions.
FIG. 13 shows a comparison of a DHT before gesture augmentation and a DHT after gesture augmentation is completed in a gesture augmentation method according to one or more embodiments.
Figure 14 is a diagram illustrating a process of producing a lecture video using DHT according to one or more embodiments.
FIG. 15 is a block diagram of a system for producing a lecture video using DHT according to one or more embodiments.
FIG. 16 illustrates a playback system for a lecture video manufactured according to one or more embodiments.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. However, the embodiments of the present invention may be modified in various different forms, and the scope of the present invention should not be construed as being limited by the embodiments described below. It is preferable that the embodiments of the present invention be construed as being provided to more completely explain the present invention to a person having average knowledge in the art. Like symbols represent like elements throughout. Furthermore, various elements and areas in the drawings are schematically drawn. Therefore, the present invention is not limited by the relative sizes or intervals drawn in the attached drawings.

While terms like "first" and "second" may be used to describe various components, these components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, a first component could be referred to as a "second component," and vice versa, without departing from the scope of the present invention.

The terminology used in this application is only used to describe specific embodiments and is not intended to limit the concept of the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the expressions “comprises” or “has” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Furthermore, it is to be understood that commonly used terms, such as those defined in dictionaries, should be interpreted to have a meaning consistent with their meaning within the relevant technical context, and should not be interpreted in an overly formal sense unless explicitly defined herein.

In some embodiments, where the implementation is otherwise feasible, a particular process sequence may be performed in a different order than described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

According to one or more embodiments herein, a method and system for transplanting an instructor's external features, such as unconscious facial micro-expressions, into a virtual avatar, a Digital Human Tutor (hereinafter, DHT) are presented.

According to this, the actual instructor's gestures and facial expressions are reflected in the movement and facial expression changes of the DHT, and in particular, the instructor's external features such as eyes, eyebrows, nose, mouth, and facial shape extracted from the actual instructor's video are expressed in the DHT, and the changes in the instructor's facial expressions are also reflected in the changes in the facial expressions extracted from the instructor's face area. In addition, the instructor's gestures are recognized and reflected in the DHT, and at this time, the gestures are augmented and reflected to expand empathy. Through this, it is expected that it will be possible to give the feeling that the instructor and learner are interacting with each other in a non-face-to-face educational environment, increase the reliability of the DHT, improve the quality of communication, and contribute to improving the constraints of the educational environment.

An embodiment according to the present disclosure includes the following three-step process for generating a DHT.

Step 1: Recognize external features including facial features and gesture features including the instructor's facial shape and expression characteristics, and unconscious micro-expression data from facial expressions.

Step 2: Create a DHT that applies the recognized instructor's features.

Step 3: Customize the generated DHT.

The instructor's physical features include facial features and joint point features. Facial features are obtained from a facial image of the instructor, while joint point features can be obtained from a photograph of the entire body or upper body region, including joints.

Below, the detection of facial feature values and joint point feature values is described separately, and first, the extraction of facial feature values according to changes in facial expression from a facial image is described.

<Step 1> Recognition of the instructor's external characteristics and micro-expression data

I. Detection of the instructor's external characteristics

This process extracts the instructor's physical features from the face and half-body or full-body regions of a camera-captured instructor video. The process proceeds as follows. The facial features include the facial feature values of each element—eyebrows, eyes, nose, mouth, and chin—as shown in Table 1. These facial feature values represent the rate of change from the neutral position or size of each element.

Figure 1 is a flowchart illustrating the process of detecting instructor facial and joint features described below. Figure 2 illustrates the arrangement of landmarks defined in FACS, and Figure 3 shows the results of the facial video-face detection-facial landmark detection process described below.

i. Facial Video Acquisition

Take a picture of the instructor's upper body or full body, including the face, from a camera capable of capturing video at 30 fps or higher.

ii. Face detection

Detect the area where the instructor's face is located within the captured video image. Viola Jones' method can be applied at this stage.

iii. Facial landmark detection

Within the detected facial region, the instructor's external features are detected. These external features include the eyebrows, eyes, nose, mouth, and chin, and their locations can be detected using 68 landmarks. These landmarks can be defined and detected based on, for example, Ekman's Facial Action Coding System (FACS). Facial muscle AUs (Action Units) are defined, and external feature movements are detected based on these AUs.

Table 2 below describes the AUs that define the movements of facial muscles for judging changes in facial expressions and the landmarks belonging to each AU.

iv. Joint extraction

This step is performed in parallel with the facial appearance feature detection process described above.

The coordinates of the instructor's joints (keypoints) are extracted from a full-body or half-body image of the instructor from which facial features have been extracted.

The human body has approximately 18 joints, and extracting the coordinates of these 18 joints is desirable for more natural gesture expression. Various methods can be used to extract these joint coordinates, and deep learning models based on machine learning can be applied. Known deep learning models include cmu, mobilenet_thin, mobilenet_v2_large, mobilenet_v2_small, tf-pose-estimation, openpose, and vnect.

In this step, the joint coordinates that have moved according to the movement of the person in the actual person image are extracted at regular frame intervals.

At this stage, the extraction of joint coordinates is performed using deep learning models such as cmu, mobilenet_thin, mobilenet_v2_large, mobilenet_v2_small, tf-pose-estimation, openpose, and vnect as described above.

The joint coordinates obtained as described above are augmented. The augmentation step of the process coordinates is a coordinate augmentation step for gesture augmentation that greatly emphasizes the instructor's gestures. The augmentation of the joint coordinates can be performed on the originally extracted two-dimensional coordinates.

II. Extraction of unconscious facial microexpression data from heartbeats

A teacher's internal emotions or feelings can be recognized through changes in facial expressions. Facial expressions are caused by facial muscle movements that reflect internal emotions. Therefore, internal emotions can be assessed or determined by evaluating facial movements, particularly those of various AUs. However, conscious movements of the teacher, unrelated to emotions, may also appear in this process, which can act as noise in assessing internal emotions. Therefore, by removing conscious movements from the micro-movements of the teacher's facial muscles, we can assess the teacher's true micro-expressions, i.e., internal emotions.

In this embodiment, conscious movements, as noise components, are filtered out. Fine movements based on internal emotions, excluding these noise components, are expressed based on a normal heart rate (BPM) ranging from 45 to 150 beats per minute (BPM).

Figure 4 is a flowchart showing the process for extracting unconscious facial micro-expressions caused by heartbeats from the facial region in an instructor video captured by a camera, and this process is described in detail below.

i. Facial Video

To detect the instructor's physical features, a facial image is acquired by capturing the instructor's upper body or full body, including the instructor's face. Acquiring the facial image involves capturing the viewer's face with a camera and continuously capturing the video content. For example, two facial images are acquired at 30 frames per second.

ii. Face detection

Face regions or points are extracted through face detection and tracking. Facial region extraction is achieved using methods such as the Viola-Jones algorithm and HOG (Histogram of Oriented Gradients), which utilize the luminance characteristics of each facial region.

iii. Area Selection

In the detected face area, the forehead and nose areas with the least noise signals are selected.

iv. Feature Extraction:

Multiple landmark points are extracted from the selected forehead and nose areas for tracking against other points. The Good-Feature-To-Track (GFTT) algorithm or the Facial Landmark Detection (FLD) algorithm can be applied to extract these landmark points. In this embodiment, the GFTT algorithm is applied to extract multiple landmark points.

v. Feature Tracking

Obtain the movement data of each extracted feature point. For this feature tracking, the KLT (Kanade-Lucas-Tomasi) tracking algorithm, the TM (Transformation Matrix) based tracking algorithm, etc. can be applied. In this embodiment, for consecutive frames, the KLT algorithm is used to track the value of the movement of the y-coordinate value in the current frame compared to the previous frame for each feature point, thereby extracting unconscious micro expression data (Micro Expression Data, MED) due to heartbeat. The sliding window technique can be used to extract the micro expression data, and the window size can be set to 30s and the interval size can be set to 1s.

III. Heartbeat signal assessment

Figure 5 is a flowchart of a process for determining a heartbeat signal, including a sliding window technique for the aforementioned micro-expression data (MED).

This process is to extract only components by cardiac responses without noise for micro-expressions from the unconscious micro-expression data (MED) extracted in the above process.

i. Bandpass Filter

For the unconscious micro-expression signals of the face, the Butterworth Bandpass Filter (5 order, 0.75-5Hz) is used to extract only the frequency band corresponding to the heart rate band of 0.75Hz (45bpm) to 2.5Hz (150bpm).

ii. Principal Component Analysis

PCA ( Principal Component Analysis ) is a process for extracting a single facial micro-expression data with the same components from the facial micro-expression data extracted from each landmark. Five components are extracted through principal component analysis. For each component, the component with the highest periodicity is extracted as the final facial micro-expression data, taking advantage of the periodic nature of biosignals. The periodicity is calculated as follows:

Here, s is a time series signal, FFT is a Fourier analysis method for converting a time series signal into a frequency band, and PS is the power spectrum of the frequency band for the time series signal s .

Here, Max Power is the largest power value in the entire power spectrum.

Here, Total Power is the sum of the entire power spectrum.

Finally, the periodicity of the time series signal s is calculated as follows.

Finally, the periodicity (heart rate) of the cardiac response is analyzed from the micro-representation of the instructor's face, and this value is used as an input value for DHT to make a micro-representation of the instructor's face.

Specifically, the facial micro-signals in the 0.75 Hz (45 bpm) to 2.5 Hz (150 bpm) band generated through PCA are reflected as amplitude values in the y-coordinates of the facial landmark feature points (eyebrows, eyes, nose, mouth, and chin) of the digital human tutor. Therefore, the input values of the DHT dynamically change the external facial expression and the internal expression imbued with internal emotions.

<Step 2> Creating a Digital Human Tutor (DHT) by Applying the Characteristics of the Recognized Instructor

DHT uses the instructor's feature values recognized in <Step 1> as default values and proceeds as follows. The instructor's feature values are related to the external facial expression and the internal expression that contains internal emotions.

I. Creation of a Digital Human Tutor (DHT)

<Step 1> Initializes the DHT using a human model from an automated software (e.g., REALLUSION Character Creator3) that projects the frontal image used to recognize the instructor's characteristics and creates the virtual avatar's external skeleton. This human model can be provided as an API or DLL (dynamic link library), making it easy to port to applications that utilize the DHT.

II. Digital Human Tutor (DHT) Appearance Correction

The DHT's appearance is normalized based on the detailed, recognized instructor's feature values (eyebrows, eyes, nose, mouth, and chin). As mentioned above, a commercially available virtual avatar can be used as the DHT, and the DHT's basic external attributes are set based on the data obtained through the aforementioned process.

III. Application of instructor characteristics

By generating a frame according to time, the amplitude value of the y (vertical direction) coordinate of the facial landmark feature point of the DHT is changed according to the periodic vibration frequency corresponding to the instructor's heart rate band recognized in <Step 1>. The application of these instructor features activates the DHT by replicating the instructor's facial expressions and emotions in the DHT, and thus the instructor's gestures, facial expressions, and complex emotional movements revealed on the face can be conveyed to the learner through the activated DHT.

<Step 3> Customizing the generated digital human tutor (DHT)

The appearance of the DHT, formed from default values in the previous step, can be freely customized by the user. The customizable feature values are listed in Table 1, and can be adjusted within a predetermined range based on the neutral feature value, or within a range of -100 to +100 for Character Creator.

Through this process, the instructor's actual facial expressions and facial emotions can be reflected in the digital human tutor's facial expressions, enabling more effective information or emotional communication.

In addition, by reflecting the joint coordinates according to the instructor's gesture changes in the above DHT, the instructor's emotions as well as the learner's interest are further strengthened.

Figure 6 is a flowchart of a DHT gesture augmentation method, and the DHT gesture augmentation method is explained with reference to this.

<Step S61>

The DHT used here has an appearance that reflects facial features obtained from the instructor using the aforementioned method. While the DHT's appearance can be artificially created, in the present disclosure, the DHT character generator captures an actual person's face, extracts facial features from it, and then reflects these as the DHT's appearance characteristics. A program called "Character Creator" can be used as an actual character generator for this purpose.

<Step S62>

Take continuous shots of the full body or half body of the above real person.

<Step S63>

Extract joint coordinates (keypoints) from full-body or half-body images of real people. The human body has approximately 18 joints, and extracting the coordinates of these 18 joints is desirable for more natural gesture expression. Various methods can be used to extract these joint coordinates, and deep learning models based on machine learning can be applied. Known deep learning models include cmu, mobilenet_thin, mobilenet_v2_large, mobilenet_v2_small, tf-pose-estimation, openpose, and vnect. In this case, joint coordinate extraction is the process of matching the DHT model to the joints of a real person.

<Step S64>

In this step, the joint coordinates extracted in the above process are mapped 1:1 to the joints of the DHT generated in the above process. That is, the coordinates of the joints of the DH are 1:1 correlated with the joint coordinates of the actual person.

<Step S65>

In this step, the joint coordinates that have moved according to the movement of the person in the actual person image are extracted at regular frame intervals.

Here, the extraction of joint coordinates is performed using deep learning models such as cmu, mobilenet_thin, mobilenet_v2_large, mobilenet_v2_small, tf-pose-estimation, openpose, and vnect as described above.

<Step S66>

This step is a coordinate augmentation step for gesture augmentation that greatly emphasizes the gestures of a real person. The increase or decrease of joint coordinates can be performed on the originally extracted two-dimensional coordinates. According to another embodiment of the present disclosure, in order to implement a more realistic active gesture, the two-dimensional coordinates (x, y) are converted into three-dimensional coordinates (x, y, z). The two-dimensional coordinates (x, y) are coordinates on a two-dimensional video image plane, and the third coordinate “z” added thereto is a coordinate in a direction perpendicular to the video image plane. By this transformation, a three-dimensional coordinate expressed as (x, y, z) can be configured by adding a coordinate in the z direction to the originally extracted two-dimensional coordinates (x, y). Here, the coordinates may include a specific region of the body, for example, a hand region, and the position of the hand may be changed up and down, left and right, or forward and backward by the coordinate transformation.

3D pose estimation can be applied to these 3D transformations, and algorithms for these transformations include Mutual PnP and Lifting from the Deep (Denis Tome, Chris Russell, Lourdes Agapito, 2017).

The number of the above 3D joint coordinates increases from the number of input 2D coordinates, which is 18, and can reach, for example, a maximum of 54 3D joint coordinates. The augmentation of the gesture at this time may include an increase or decrease in the coordinates or an augmentation of the angle on the coordinates.

<Step S67>

Augmented coordinates, for example, augmented 2D or 3D joint coordinates, are applied to the DHT model through the above process to implement the augmented gesture in the DHT.

<Step S68>

A DHT having the above-described augmented gesture is implemented and activated in a target video, and at the same time, a professor during a lecture is continuously filmed to detect a change in the gesture of the next real person, and the routine is repeated by returning to the above-described <Step S65> to implement the DHT having the augmented gesture.

In summary, the main process of the present disclosure is to initially create a DHT object, map the joint characteristics of this DHT object to the joints of an actual person, perform DHT initialization, and then continuously recognize the joint coordinates of an actual person, augment them, and then reflect them in the DHT object to activate them.

The joints mentioned in this disclosure are classified into 18 joints as shown in FIG. 7.

Referring to Figure 7, the maximum number of joints extracted from an actual person is 18, which includes not only limb and shoulder joints, but also the nose, both eyes, both ears, mouth, and neck of the face.

For a more natural self- or gesture-implementation in the above joints, it is necessary to use all joints.

In the following, we describe the DHT of the augmented gestures that were actually implemented.

Figure 8 illustrates a tutor, or DHT (Digital Human Tutor), who guides learning through video. In the video of Figure 8, the DHT passively positions both hands inside the upper body.

Figure 9 shows an example of augmenting some of DH's gestures, augmenting the angle in the x direction in 3D coordinates.

As can be seen by comparing Figures 8 and 9, Figure 9 shows more active and lively hand movements than Figure 8.

Fig. 9 shows an example of augmenting the angle in the x direction in the 3D coordinates of DHT, Fig. 10 shows an example of augmenting the angle in the y direction, and Fig. 11 shows augmentation in the z direction. And Fig. 12 shows augmentation in all directions x, y, and z.

Figure 13 compares the DHT before augmentation (left) and the DHT with coordinate angle augmentation in all x, y, and z directions (right).

As shown in Figure 13, the post-augmentation posture appears more active and dynamic than before. This demonstrates that DH's nonverbal expressions are very strongly expressed.

Various video controllers in the form of programs can be used to convert the above-mentioned video. For example, a software called Unity can be used.

In Unity, the movement of each joint can be augmented with a range value between 0 and 10 in the slider UI provided in Unity, and the augmentation of each joint can increase or decrease the joint angle value within a certain range, for example, from a maximum of 50 degrees to a range of -50 degrees. As shown in FIGS. 6 to 9, when augmentation of an arm gesture is desired, the image processor selects the joint corresponding to the arm, selects the x, y, and z angles of the joint, and augments the angle values with a range value between 0 and 10.

The DHT generated by the above method can be applied to various fields, including as a DHT in video learning systems. In video learning systems, the instructor's emotional and verbal expressions, as well as nonverbal expressions expressed through body gestures, can be effectively conveyed to learners, thereby enhancing learning efficiency. This nonverbal communication can also be effectively utilized in virtual worlds.

Figure 14 is a diagram illustrating the process of producing a lecture video using DHT as described above, and Figure 15 schematically shows the structure of the system for this.

Referring to Figures 14 and 15, two cameras (31a, 31b) are required to produce a lecture video.

One camera (31a) captures the face of an actual professor, passes it through an image processor (31), and then creates a DHT resembling the professor's appearance using a character generator (33) through the process described above.

And, another camera (31b) records the lecture content by the professor, and from this, through the process described above, the face and joints are detected by the image processor (32), and changes in the professor's facial expression, gaze, gestures, etc. are detected by the characteristic processor (34), and characteristic value variables are measured or extracted.

The DHT model is activated by substituting or transplanting the above characteristic value variables into the DHT model obtained through the above process.

The DHT model is activated by the DHT model feature control unit (35), which implants the instructor's facial expressions, gaze, and gestures into the DHT model in addition to the instructor's appearance. The lecture video generated through this process is stored on a medium and distributed through the medium.

FIG. 16 schematically illustrates a lecture system (1) for taking an online lecture using the lecture video according to one embodiment of the present disclosure.

The above lecture system (1) downloads the above lecture video material or plays it through streaming, and transmits it to the learner (20) through the display (12). Since the lecture video presented to the learner (20) through the display (12) mostly includes audio components, an audio device for playing the audio components may be additionally added to the lecture system (1). Since the above lecture system (1) is based on a general computer, it has a hardware structure based on a computer system (1) including input/output devices such as a keyboard (14), a mouse (15), and a monitor (12) that are basically installed in a computer, and a main body (11) to which they are connected.

While exemplary embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications and variations can be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims. Therefore, modifications to future embodiments of the present invention will not depart from the scope of the present invention.

Claims (18)

A step of acquiring a video of the actual lecturer and facial image through a camera;
A step of extracting the instructor's joint (keypoint) coordinates and facial appearance features from the lecture video and facial image by an image processor;
A step of determining an augmentation weight for the change in the external feature and the value of the joint coordinate from the external feature of the face by a characteristic processor;
A step of generating a digital human tutor corresponding to the instructor's facial image using the instructor's facial feature image by the character generation unit; and
A step of reflecting the external characteristics of the instructor and joint coordinates according to the change in gestures to the digital human tutor by the character control unit, and changing the joint coordinates of the digital human tutor by the joint coordinates augmented by the augmented weights to activate the digital human tutor of the augmented gesture compared to the instructor's gesture;
Here,
In the step of extracting the external characteristics of the above professor,
Extract the instructor's characteristic points from the above lecture video, extract the movement data of the extracted characteristic points, and extract the fine expression data from the movement data.
In order to extract unconscious micro-expression data from the above micro-expression data,
A method for creating a gesture-augmented digital human tutor, characterized in that filtering of a predetermined frequency is performed on the above micro-expression data, the periodicity of the heartbeat is determined by principal component analysis (PCA) on the filtered micro-expression data, and the periodicity is used as an input value for the micro-expression of the digital human tutor. In the first paragraph,
Step of extracting the above joint coordinates:
A step of extracting joint coordinates on a two-dimensional plane by an image analysis unit; and
A method for creating a gesture-augmented digital human tutor, comprising: a step of extracting three-dimensional joint coordinates (x, y, z) by inferring a third direction (z) perpendicular to the two-dimensional plane using a 3D analyzer; In the second paragraph,
In the step of determining the above augmentation weight,
A method for creating a gesture-augmented digital human tutor, wherein a weight is determined for at least one joint coordinate among three coordinates x, y, and z in the above three-dimensional joint coordinates.
delete In the first paragraph,
A method for creating a gesture-augmented digital human tutor, which applies a KLT (Kanade-Lucas-Tomasi) tracking algorithm or a TM (Transformation Matrix)-based tracking algorithm to track the movement of the feature points in order to extract the above-mentioned micro-expression data.
delete delete In any one of paragraphs 1, 2, 3 or 5,
The above-mentioned instructor's external features are extracted as landmarks defined in FACS, and a method for creating a gesture-augmented digital human tutor reflecting the external features in units of AUs based on the landmarks for the above-mentioned digital human tutor. One or more cameras that capture video of the actual instructor's lecture and facial images;
An image processor that extracts the actual instructor's joint (keypoint) coordinates and facial features from the above lecture video and facial image;
A feature processor that extracts changes in facial external features from a facial image during a lecture by the instructor and changes in joint coordinates from the lecture video, and determines augmentation weights for the values of the joint coordinates, extracts feature points of the instructor from the lecture video, extracts movement data of the extracted feature points, extracts micro-expression data from the movement data, performs filtering of a predetermined frequency on the micro-expression data to extract unconscious micro-expression data from the micro-expression data, and determines the periodicity of a heartbeat used as an input value for micro-expression of a digital human tutor by principal component analysis (PCA) on the filtered micro-expression data;
A character generation unit that uses the facial image of the instructor to generate the digital human tutor corresponding to the facial image of the instructor;
A character control unit that reflects the above-mentioned external characteristics to the digital human tutor, and activates the digital human tutor to change the joint coordinates of the digital human tutor by the joint coordinates augmented by the augmented weight; and
A system for generating a gesture-augmented digital human tutor, comprising a lecture video generation unit that generates a lecture video including the digital human tutor. In paragraph 9,
A system for generating a gesture-augmented digital human tutor, wherein the character generation unit extracts feature points from the facial image in the step of extracting external features and sets the appearance of the digital human tutor using the feature points.
delete In Article 10,
A character generation system for generating a digital human tutor that selects landmarks defined in FACS as the above-mentioned characteristic points.
delete In Article 10,
The above-mentioned feature processor is a system for generating a gesture-augmented digital human tutor, which applies a KLT (Kanade-Lucas-Tomasi) tracking algorithm or a TM (Transformation Matrix)-based tracking algorithm to motion tracking for the feature points in order to extract the micro-expression data.
delete In paragraph 9,
The above characteristic processor extracts landmarks defined in FACS as the external features of the instructor, and a gesture-augmented digital human tutor generation system that reflects the external features in units of facial muscle AU (Action Unit) by the landmarks for the digital human tutor. In paragraph 9,
The above characteristic processor is,
The joint coordinates on a two-dimensional plane are extracted by the image analysis unit, and
A system for generating a gesture-augmented digital human tutor that extracts three-dimensional joint coordinates (x, y, z) by inferring a third direction (z) perpendicular to the two-dimensional plane using a 3D analyzer. In Article 17,
A system for generating a gesture-augmented digital human tutor, wherein the above-mentioned characteristic processor determines an augmentation weight for at least one joint coordinate among three coordinates of x, y, and z in the three-dimensional joint coordinates.