KR101492816B1 - Apparatus and method for providing auto lip-synch in animation - Google Patents

Abstract

An animation lip sync automation apparatus and method are disclosed. An audio file corresponding to a scenario of the animation image and synchronized with a frame constituting the animation image is input to the audio input unit. The speech recognition unit recognizes the height of consonants, vowels, and sounds constituting each associate from the speech file and outputs the speech recognition result. The mouth shape selection unit determines a pronunciation classification corresponding to each of the associations on the basis of the speech recognition result, and determines a mouth shape corresponding to the determined pronunciation classification by a predetermined number of mouth shapes considering the pronunciation characteristics of Korean Select from the shape store. The mouth shape selection unit selects a first mouth shape with both lips attached to the mouth shape of the silent section and selects a second mouth shape with both lips spaced apart by a first size before the first number of frames at which the speech starts A mouth shape corresponding to a pronunciation classification determined for each of the associations is selected from a point of time when a voice is started and a final mouth shape is maintained for a second number of frames from a point of time when a voice is terminated, The first mouth shape is selected. According to the present invention, the mouth shape of the animation character can be perfectly matched with the pronunciation of the voice actor, and lip-synching of the animation character can be automatically implemented in real time by accurately reflecting the pronunciation structure of Korean and the production characteristics of the animation.

Description

Technical Field [0001] The present invention relates to an animation lip-sync automation apparatus and method,

The present invention relates to an apparatus and method for animating lip syncs, and more particularly, to a method and apparatus for automatically synchronizing animations of animated lip images produced by a computer graphics and a voice of a voice actor, To a mouth shape corresponding to the shape of the mouth.

The animated film produced by the major movie production companies in the United States improves the immersion of the audience by perfectly reproducing the character 's performance with the perfect lip - sync implementation. Thus, by expressing the mouth shape of the character in the animation movie the same as the mouth shape corresponding to the voice of the voice actor, American animation films are called high quality animation. However, there is absolutely lack of research and technology in lip - syncing in Korean animation industry and industry.

Generally, lip-syncing of an animation is performed by first creating an animation, then recording a script while a viewer watches the image, or by an experienced worker in a mouth-shaped database preset to correspond to the actual pronunciation of each word of the script After choosing the shape of the mouth of the animation character to produce the animation, Sung Woo is taking the method of recording the script while watching the video. However, since the former method does not accurately reflect the mouth shape corresponding to the actual pronunciation of the person, the reality is lowered and the latter method must be performed by a worker having a specialized knowledge and experience, so that it takes a long period of time to produce a lip- There is a problem that it can not.

On the other hand, when a person speaks, the shape of a mouth mainly changes according to the vowel "ah, a, e, o, ooo, ooo, ooo" and extracts the data of the vowel from the voice signal, However, since the influence of the consonant used as a support is eliminated, there is a problem that it is difficult to express a correct mouth shape. To cope with a mouth shape for each vowel, There is a problem in that the shape changes and the processing is not performed on the silent section, so that it differs from the mouth shape that appears when the person actually pronounces the sound.

Korean Unexamined Patent Application Publication No. 2002-0026617 (Title of the invention: Method of Providing Lip Sync Animation by Real-Time Speech Recognition, Release Date: April 12, 2002) Korean Patent Laid-Open No. 2008-0008432 (entitled " Lip Sync Synchronization Method and Apparatus for Speech Signal, Disclosure Date: Jan. 23, 2008)

Korean Society of Cartoon and Animation 2008 13th "Study on Korean Lip Sync of Animated Characters"

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide a method and apparatus for accurately characterizing a pronunciation structure of a Korean character and an animation, Apparatus and method.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide a method and apparatus for accurately characterizing a pronunciation structure of a Korean character and an animation, There is provided a computer-readable recording medium recording a program capable of causing a computer to execute the method.

According to another aspect of the present invention, there is provided an apparatus for automating an animation lip sync, including: a voice input unit for inputting a voice file synchronized with a frame constituting the animation image, corresponding to a scenario of an animation image; A voice recognition unit for recognizing the height of consonants, vowels, and sounds constituting respective associations from the voice file and outputting voice recognition results; A mouth shape storage unit in which a predetermined number of mouth shapes considering pronunciation characteristics of Korean are stored corresponding to pronunciation categories; And a mouth shape selection unit for determining a pronunciation classification corresponding to each of the associations based on the speech recognition result and selecting a mouth shape corresponding to the determined pronunciation classification in the mouth shape storage unit, , A first mouth shape with both lips attached to the mouth of the silent section is selected and a second mouth shape where both lips are separated by a first size before the first number of frames at which speech starts, The mouth shape corresponding to the pronunciation category determined for each of the associations is selected from the point of time when the voice is terminated and the final mouth shape is maintained for the second number of frames from the end of the voice, The first mouth shape is selected.

According to another aspect of the present invention, there is provided a method for automating animation lip sync according to the present invention, comprising the steps of: (a) constructing respective associations from a voice file synchronized with a frame constituting the animation image, Recognizing the height of consonants, vowels, and sounds and outputting speech recognition results; (b) A first mouth shape with both lips is selected from a mouth shape database in which a predetermined number of mouth shapes considering Korean pronunciation characteristics are stored corresponding to pronunciation categories, Selecting a second mouth shape in which both lips are separated by a first size before a first number of frames; (c) determining a pronunciation classification corresponding to each of the associations based on the speech recognition result, and selecting a mouth shape corresponding to the determined pronunciation category in the mouth shape database; And (d) selecting the first mouth shape as a mouth shape until the next voice starts after the last mouth shape is maintained during the second number of frames from the end of the voice.

According to the apparatus and method for animating lip-sync according to the present invention, it is possible to perfectly match the mouth shape of the animation character with the pronunciation of the voice act accurately by accurately reflecting the pronunciation structure of the Korean and the production characteristics of the animation, It can be implemented automatically.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a configuration of a preferred embodiment of an animation lip sync automation apparatus according to the present invention;
FIG. 2 is a diagram showing a phonetic classification and corresponding mouth shapes,
FIG. 3 is a flowchart illustrating a method of performing an animation lip sync automation method according to a preferred embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a preferred embodiment of an animation lip sync automation apparatus and method according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a view showing a configuration of a preferred embodiment of an animation lip sync automation apparatus according to the present invention.

1, an animation lip sync automation apparatus according to the present invention includes an image input unit 110, a sound input unit 120, a voice recognition unit 130, a mouth shape storage unit 140, a mouth shape selection unit 150, A shape synthesizing unit 160 and a control unit 170.

The image input unit 110 is a component that receives an animation image. An image input to the image input unit 110 differs in number of frames per second by 24, 30, etc. in accordance with usage, quality, etc. In the following description, an image composed of 30 frames per second will be described as an example.

The voice input unit 120 is a component that receives a voice file recorded by a voice actor in accordance with a scenario of an animation. The voice file input to the voice input unit 120 is synchronized with each frame of the animation image, and thus the voice file is a digital file set at 30 frames per second.

The voice recognition unit 130 analyzes the voice file inputted through the voice input unit 120, and recognizes the consonants, vowels, and the sound levels of the respective associations. The speech recognition unit 130 outputs a speech recognition result. At this time, the speech recognition result may be output as a code corresponding to each of the associations (for example, a complete Hangul code, a combination Hangul code, etc.).

The mouth shape storage unit 140 stores a certain number (for example, eight) of mouth shapes in consideration of pronunciation characteristics of Korean corresponding to the pronunciation category. The pronunciation classification is obtained by classifying the actual mouth shape based on the mouth shape stored in the mouth shape storage unit 140 when each of the associations is pronounced. In Fig. 2, phonetic classification and corresponding mouth shapes are shown.

The mouth shape selection unit 150 determines a pronunciation classification corresponding to each of the associations based on the speech recognition result input from the speech recognition unit 130. [ Table 1 shows examples of pronunciation classification criteria.

Pronunciation classification Pronunciation classification A Umm,,,,, ㅍ B ㅅ,,, ㅊ, ㅡ, ㅣ, 쓰 C A, ㅓ, ㅔ, ㅐ D A (long sound, high sound), ㅑ (long sound, high sound) E ㅓ, ㅕ F ㅗ, ㅛ G ㅜ, ㅠ H C, d,

Table 1 shows eight phonetic categories, but it is possible to add phonetic classifications as necessary, and the phonetic classifiers can be further finely structured.

After determining the pronunciation classification corresponding to each of the associations, the mouth shape selection unit 150 selects the mouth shape according to the following criterion.

1. Select mouth shape C before the first number of frames (for example, 10 frames) at which voice starts.

2. From the beginning of the speech, the mouth and the sound input in the second number of frames (for example, four frames) are analyzed and the mouth shape corresponding to the corresponding position is selected. The reason why the mouth shape is selected every four frames is that the purpose of reducing the amount of calculation and the recognition ability of the human visual change (i.e., even if the mouth shape is maintained for a certain period of time, will be. At this time, the speech recognition unit 130 may output speech recognition results every four frames.

3. Select the mouth shape A before the third number of frames (for example, 2 frames) only when the lips close sound, such as ㅁ, ㅂ, In this case, the mouth should be selected as the vowel at the moment of sound, but the mouth shape A is selected in the previous 2 frames.

4. In the silence between the sentence and the sentence, keep a fourth number of frames (for example, 10 frames) in the mouth shape just before the silence, and change to the mouth shape C until the next voice comes out.

The mouth shape combining unit 160 combines the mouth shape selected by the mouth shape selecting unit 150 with the animation image input from the image input unit 110 to generate an image in which voice and mouth shapes are synchronized. At this time, the mouth composition unit 160 receives the information (for example, frame number) of the selected image frame by the mouth shape selection unit 150, and the mouth shape of the mouth of the character in the frame constituting the animation image And detects a corresponding region and synthesizes the mouth shape selected by the mouth shape selecting unit 150 in the corresponding region. The mouth shape combining unit 160 may be implemented as a separate apparatus.

The control unit 170 controls the operation of the next component, and receives a set value from the user. The set values received from the user may be the first to fourth values.

FIG. 3 is a flowchart illustrating a method of performing an animation lip sync automation method according to a preferred embodiment of the present invention.

Referring to FIG. 3, the speech recognition unit 130 analyzes a speech file inputted through the speech input unit 120, and recognizes the consonants, vowels, and sounds constituting the respective associations, and outputs speech recognition results (S300). Next, the mouth shape selection unit 150 determines a pronunciation classification corresponding to each of the associations based on the speech recognition result input from the speech recognition unit 130 (S310). Next, the mouth shape selecting unit 150 selects the mouth shape A as the mouth shape of the silent section in the mouth shape storage unit 140, and selects the mouth shape C (S320). Next, the mouth shape selecting unit 150 selects a mouth shape corresponding to the pronunciation classification determined for each of the associations in the mouth shape storage unit 140 (S330). At this time, the mouth shape selection unit 150 selects the mouth shape A 2 frames before the lips are closed, such as ㅁ, ㅂ, ⋯, etc., and selects the mouth shape according to the vowel at the moment of sounding. Next, the mouth shape selecting unit 150 keeps the final mouth shape for about 10 frames from the point where the silence section starts (i.e., the point at which the voice ends), and changes the mouth shape until the next voice starts The mouth shape C is selected in the mouth shape storage unit 140 (S340). Next, the mouth shape combining unit 160 combines the mouth shape selected by the mouth shape selecting unit 150 with the animation image input from the image input unit 110 to generate an image in which voice and mouth shapes are synchronized (S350 ).

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may be implemented in the form of a carrier wave (for example, transmission via the Internet) . The computer-readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation in the embodiment in which said invention is directed. It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the appended claims.

Claims (8)

A voice input unit for inputting a voice file synchronized with a frame constituting the animation image, corresponding to a scenario of the animation video;
A voice recognition unit for recognizing the height of consonants, vowels, and sounds constituting respective associations from the voice file and outputting voice recognition results;
A mouth shape storage unit in which a predetermined number of mouth shapes considering pronunciation characteristics of Korean are stored corresponding to pronunciation categories; And
And a mouth shape selection unit for determining a pronunciation classification corresponding to each of the associations based on the speech recognition result and selecting a mouth shape corresponding to the determined pronunciation category in the mouth shape storage unit,
The mouth shape selecting unit selects a first mouth shape with both lips attached to the mouth shape of the silent section and selects a second mouth shape with both lips being separated by a first size before the first number of frames A mouth shape corresponding to a pronunciation classification determined for each of the associations is selected from a point of time when a voice is started and a final mouth shape is maintained for a second number of frames from a point of time when a voice is terminated, The first mouth shape is selected as the mouth shape from the beginning of the speech, and a pronunciation classification corresponding to each of the associations is determined on the basis of the speech recognition result for every third number of frames from the start of speech, Wherein the first lip shape is selected from the mouth shape storage portion and the first lip shape is selected before a fourth number of frames when a lips sound is heard. Automation devices. The method according to claim 1,
An image input unit into which the animation image is input; And
Further comprising a mouth shape synthesizing unit synthesizing the mouth shape selected by the mouth shape selecting unit with an animation image input through the image input unit to generate an image in which voice and mouth shapes are synchronized. delete delete (a) outputting a speech recognition result by grasping a height of a consonant, a vowel, and a sound constituting each association from a speech file corresponding to a scenario of the animation image and synchronized with a frame constituting the animation image;
(b) A first mouth shape with both lips is selected from a mouth shape database in which a predetermined number of mouth shapes considering Korean pronunciation characteristics are stored corresponding to pronunciation categories, Selecting a second mouth shape in which both lips are separated by a first dimension before a first number of frames;
(c) determining a pronunciation classification corresponding to each of the associations based on the speech recognition result, and selecting a mouth shape corresponding to the determined pronunciation category in the mouth shape database; And
(d) selecting the first mouth shape as a mouth shape until the next voice starts after a final mouth shape is maintained during a second number of frames from a time when a voice is terminated,
In the step (c), a pronunciation classification corresponding to each of the associations is determined on the basis of the speech recognition result for each of the third number of frames from the start of speech, and a mouth shape corresponding to the determined pronunciation classification is selected To automate the animation lip sync. 6. The method of claim 5,
(e) synthesizing the selected mouth shape with the animation image to generate an image in which voice and mouth shapes are synchronized with each other. delete A computer-readable recording medium storing a program for causing a computer to execute the animation lip sync automation method according to claim 5 or 6.

Images