JP2020072415A - Video/audio synthesis method - Google Patents

Abstract

To provide a video/audio synthesis method that can reuse a set once produced and easily produce a video content in which an appearing character is performed by people from each country while reducing the production cost without using conventional subtitling or voice translation technology.SOLUTION: A video/audio synthesis method includes performing video synthesis of synthesizing a live-action person video with a stage video that is the background of a person (S12), synthesizing a recorded voice and a sound that becomes the background of the recorded voice (S11), producing a video content by synthesizing video synthesis output and sound synthesis output (S13), replacing the live-action person video with a live-action person video of a person different from the live-action person video (S20), performing video synthesis of synthesizing a live-action person and a stage video of the different person (S23), replacing the recorded sound with of a recorded sound of the person different from the recorded sound of the video content (S20), performing sound synthesis of synthesizing sound (S25), and producing a video translation content by synthesizing video synthesis output and sound synthesis output of a different person (S26).SELECTED DRAWING: Figure 4

Description

  The present invention relates to a video / audio synthesizing method particularly related to movie drama.

In the past, when foreign movie contents imported from Japan to Japan or Japanese movie contents exported to foreign countries are screened, broadcasted and distributed online, in order to make the drama content understand to the viewers of the other country. There are two video translation techniques. One is to insert subtitles translated into the language of the other country for each scene at the bottom of the screen so that the contents of the work can be understood by viewing the video and subtitles. The other is that the voice of the person who appears in the drama is spoken by the voice actor of the partner country (person with only voice), and the voice is replaced by the recorded voice so that the viewer understands the content of the work.
The existence of such two means of video subtitles or dialogue dubbing makes it easy to export and sell video contents of foreign dramas in which foreign characters appear worldwide.

  When viewing the video content of such a foreign movie, for example, as shown in Patent Document 1, when the language used in the movie and the language in the region to be broadcast are different, based on the language information of the video content. There is disclosed an information processing apparatus capable of outputting a translated voice in a language of a region to be broadcast after voice recognition, and capable of converting the translated voice data into character information and displaying it as a subtitle or the like. As a result, foreign movie contents can be easily viewed and understood regardless of the language ability of the viewer.

JP, 2003-16080, A (paragraph 0070-paragraph 0074).

However, even if the audio translation output and the video subtitle are mechanically realized based on the language information of the video content as in Patent Document 1, the viewer does not change the live voice of the person or the realistic nuance. I cannot receive it directly.
In addition, when a Japanese movie of the same content is used as a foreign movie or a foreign movie is used as a Japanese movie, it is necessary to prepare all the sets to be staged in the countries in which they are produced and aired.

  Therefore, according to the present invention, in order to solve such a problem, a set once produced can be reused without using a conventional subtitle creation or voice translation technique, and a person appearing in each country can be broadcast. The present invention provides a video / audio synthesis method for easily producing video contents played by a person while reducing production costs.

In order to achieve such an object, a video / audio synthesizing method for synthesizing video and audio of the first invention is
A step of image composition for associating and synthesizing a live-action person image actually taken by a camera and a stage image which is a background of a person produced by computer graphic,
A sound synthesizing step of synthesizing the recorded voice of the photographed person and a sound sound which is a background of the recorded voice in association with each other;
Furthermore, a step of synthesizing the output of the image synthesis step and the output of the sound synthesis step to produce a video content,
Replacing with a live-action portrait image of a person different from the live-action portrait image of the video content;
An image synthesizing step of synthesizing a live-action portrait image of a person different from the live-action portrait image and the stage video in association with each other,
Replacing the recorded sound of the video content with the recorded sound of a person, and synthesizing the sound by associating with the sound,
Further, it is characterized in that it comprises a step of producing an image translated content by synthesizing the output of the image synthesis step of the different person and the output of the sound synthesis step.
A second aspect of the invention is characterized in that, in the first aspect of the invention, the synthesis of the live-action person image and the recorded voice and the synthesis of the live-action person image and the recorded voice of different persons are performed in synchronization.

  According to the present invention, the set once produced can be reused without using the conventional captioning or voice translation technology, and the production cost of the video content in which the characters appearing are played by the characters of the broadcasting countries. The ease of production while reducing the number allows the viewer to recognize that the image is produced in his / her own country, and improve the user's liking, empathy and related consumption.

It is a block diagram of a configuration of an embodiment of the present invention. It is the whole flow figure to the video content production of this execution form. (A) is a format explanatory diagram of a live-action portrait video and a recorded voice of the Japanese version, (b) is a format explanatory diagram for synthesizing a live-action portrait video and a CG stage video, and (c) is a recorded voice and SE. It is a format explanatory drawing which synthesize | combines BGM, (d) is a format explanatory drawing which synthesize | combines (b) and (c). FIG. 4 is a flow chart of a main part of producing video translated content according to the present embodiment. (A) is a format explanatory diagram in which (d) of FIG. 3 of the Japanese version is replaced with the Chinese version, and (b) is a format in which (d) of the Japanese version is replaced with the US version. FIG.

  An embodiment of the present invention will be described below with reference to the accompanying drawings. The basic configuration of the present embodiment is a video / audio synthesizing method for further synthesizing a person's voice including BGM such as a sound effect to a synthetic video of a live-action video captured by a camera or the like and a CG video of a background set.

  First, a Japanese film to be broadcast in Japan, for example, a Japanese actor (hereinafter referred to as a person) makes a movie in Japanese (hereinafter referred to as video content). In the subsequent process, the person part of this video content was replaced in a foreign broadcast area, for example, a Chinese-language movie using Chinese or an American-language video / audio-translated movie (below). , Video translation content).

[Process related to production of video contents and video translated contents]
The process of synthesizing the video and the audio relating to the production of the video content will be described step by step using the flow of FIG. However, the steps up to step S6 can be said to be preparations for producing the video content, and therefore will be briefly described.
Step S1 is the creation of a script (story), which is completed by assuming images and scenes to be realized by the script, sounds such as dialogue and sound effects, and persons and the stage in which they appear.
A step S2 is a video production plan (creating a storyboard) based on the script, and the movement of the characters, the position of the camera, and the like are shown in a picture for each cut based on the script.
A step S3 creates a shooting plan video (video story) based on the storyboard.
In step S4, a shooting plan is prepared based on the contents of the video story.
In step S5, a set for composite shooting is designed.

As a next process, in step S6, after designing and producing a set for video composition shooting, a stage set is created with one color suitable for performing chroma key for video composition (using one color suitable for composition, for example, green and blue). To paint.
In step S7, a person's performance is photographed and a voice is recorded at the same time in the one-color composition stage set created in step S6. That is, it is synchronized with voice recording (this state is indicated by the dotted line indicated by the symbol T). That is, the synthesis of a live-action portrait image and a recorded voice, which will be described later, and the synthesis of a live-action portrait image of a different person and a recorded voice are performed in synchronization.
One camera or a plurality of cameras of two or more are used for photographing a person's performance. In the description of the present embodiment, the term “actual image” is appropriately used to distinguish an image such as a CG image as a term indicating an image actually taken.

The subsequent steps S8, S10, and S12 are processes relating to video processing of a real image and a CG image.
In step S8, the background of the photographed video is extracted from the video in which the person performs with the background having a color tone unrelated to the person who is the subject. That is, only the person is cut out, and the color tone is extracted optically or electronically and eliminated.
In step 10, the known technique of chroma key composition is used in which the background image of the captured image is extracted and the CG image is replaced and combined. In other words, the image of the person is inserted into the background of the person in association with the one-color stage set.
In step 12, the photographed person image and the CG image are associated and combined.

The following steps S9 and S11 are processes relating to sound processing of voice and BGM such as sound effects. In addition, in the description of the present embodiment, the term “actual recording” is appropriately used in order to distinguish BGM such as sound effects as a term indicating a voice of an actor actually recorded.
In step S9, BGM such as SE of sound effects and music is created based on the script and storyboard in steps S1 and S2.
Step 11 is a sound process of synthesizing voice and SE / BGM in association with each other. SE is an abbreviation for Sound Effect (sound effect) and BGM is an abbreviation for Back Ground Music (background music).

  In step 13, the video image synthesized in step 12 and the sound synthesized in step S11 are associated with each other and further comprehensively synthesized to complete the video content.

Next, a process of substituting the Japanese of the video content produced by the above series of processing with a foreigner (for example, a person from China or the United States) to produce the video translated content will be described with reference to FIG.
The steps S7, S8, S10 of the process relating to the actual recording of the image in FIG. 2, the steps S9, S11 relating to the actual recording of the sound, the image / voice synthesis output step S13 are the steps S20, S21 of the process relating to the actual recording of the image in FIG. , S22, steps S24 and S25 relating to actual recording of sound, and video / audio synthesis output step S26, respectively, and perform similar processing. That is, as indicated by the dotted frame in step S20, a foreign person plays along with the same script in a composition set (stage) played by a Japanese person, and a live-action and voice recording of the person are performed synchronously. Be done.
In this way, when producing video content where the characters who appear are in the scenes of people in different countries, it is possible to reuse the composition set played by the Japanese people, and it is easy to produce while reducing production costs. can do.

Specifically, in step S7 (S20), as shown in FIG. 3A, the voice recording data voice JPN001 and the video image data JPN001 share synthesis information, but on the time axis t. It is tied along. "JPN001", "JPN002", etc., which are commonly added to the audio JPN001 of the actual recording and the actual reproduction JPN001 of the actual image. ． ． ． Is a synthesis information related to control from the video / sound synthesis control unit 10 which will be described later, and serves as a key for cutting and pasting and editing the video material and the audio material for the Japanese at a necessary position. As a result, it is possible to easily manufacture the image and sound by a synthesis process without requiring complicated control and operation.
Further, if it is the synthetic information about Chinese, it is the live-action CHN001 and voice CHN001, and if it is the synthetic information about the United States, it is the live-action USA001 and voice USA001.
In the description of the present embodiment, information for synthesizing video data and CG data in association with each other, information for synthesizing recorded sound and SE / BGM in association with each other, and all of these data will be described. The information to be associated and combined is simply used as combined information.

  In step S12 (S23), as shown in FIG. 3B, starting from the frame information JPN001, which is the composition information, the real image JPN001 of the live image data and the CG data CGJPN001 of the CG data are sequentially combined along the time axis t. And keep it.

  In step S11 (S25), as shown in FIG. 3C, with the frame information JPN001 as the synthesis information as a starting point, the voice JPN001 of the human voice data and the SE / BGM data SE / BGMJPN001 are set along the time axis t. Sequentially synthesized and kept.

  Finally, in step S13 (S26), as shown in FIG. 3D, starting from the frame information JPN001 which is the composition information, the real image JPN001 of the real image data combined in step S12 and the CG data CGJPN001 of the CG data. Then, the voice JPN001 of the human voice data and the SE / BGM data SE / BGMJPN001 synthesized in step S11 are sequentially synthesized and kept along the time axis t, and a video content (video translation content) is produced.

  In addition, in the processes of steps S1 to S6, the CG images and SE / BGM materials produced by the pre-processing process common to the Japanese version of the video content and the foreign version of the video translated content can be used as they are, so that they are duplicated. The steps S1 to S6 of FIG. 2 can be omitted.

  As described above, a step S12 of image synthesis for synthesizing a live-action person image and a stage image which is a background of the person in association with each other, a step S11 of synthesizing the recorded voice and a sound synthesizing the background of the recorded voice, and further the image. Step S13 of producing the video content by synthesizing the synthetic output and the sound synthetic output, step S20 of replacing the actual person image with the actual person image different from the actual person image, and associating the actual person image of the different person with the stage image Step S23 of synthesizing video, step S20 of replacing the recorded voice of the video content with the recorded voice of a person, step S25 of synthesizing the sound in association with the sound, and step of synthesizing the image of a different person. And the output of the sound synthesis step are combined to produce the video translation content. In step S26, the person in the composition set in step 7 was Japanese and Japanese, but the person in the composition set in step S20 can be replaced by a foreigner with a foreign language. Foreign version video translation contents such as US version are produced.

[Video / audio synthesis method and operation function of the provided video / audio synthesizer]
Next, operation functions of the video / audio synthesizing apparatus according to the embodiment of the present invention will be described.
As shown in FIG. 1, the video / audio synthesizing device 1 is composed of a video unit 2 and a video / sound synthesis control unit 10 for controlling the sound 3.

[Video sound synthesis controller]
The video / sound synthesis control unit 10 uses a schedule table created in advance when the video content is produced, for example, characters for each scene, necessary costumes / props, and disappearances (1 Items that can be used only once, such as breaking newspapers and lit cigarettes, etc.), are written in detail, and there are things such as an incense table and a script of the script to facilitate video production. As a result, it is possible to sequentially control the operation of each part of the image section 2 and the sound 3.

[Video part]
The video unit 2 stores a CGDB (stage) 11 in which background images by computer graphics (CG) are accumulated, a CG stage image reproducing unit 12 in which the CG images are reproduced according to the person who is actually photographed, and acting images by Japanese. Data information is shared between the live-action DB (JPN) 14, the live-action personal-image playback unit 17 that reads out and plays back a person video from the live-action DB (JPN) 14, the CGDB (stage) 11, and the live-action DB (JPN). The shooting data reference DB 13, the live-action / CG synthesizing unit 18 for synthesizing the CG image played back by the CG stage video playback unit 12 and the human image read out and played back by the live-action portrait video playback unit 17, and the live-action / CG synthesis unit The image output section 19 outputs and displays the image combined in step 18.

  In addition, a live-action DB (CHN) 15 in which acting images by the Chinese are accumulated and a live-action DB (USA) 16 in which acting images by the American are accumulated are shown in FIG. 1. These are Japanese version movies. Is used when making a Chinese or American movie after it has been produced, not in the process of making a Japanese movie. The DB is an abbreviation for DataBase, and the database structure of the live-action DB (CHN) 15 and the live-action DB (USA) 16 is the same as the format and structure of the database of the live-action DB (JPN) 14, and is the same as that of Chinese or American. There is only a difference in the person.

The CGDB (stage) 11 is composed of three-dimensional (3D) CG data in a mov format, which is a background image of a person image that is chroma key-combined, and time-based (frame-based) information. For example, FIG. 3B is a diagram schematically showing a data format, in which the CG data is CGJPN00, and the frame information JPN001 which is the composition information is used as a starting point and the live-action video data JPN001 is along the time axis t. 2 shows a state of sequentially synthesizing. These synthesizing operations are performed by the synthesizing information from the video / sound synthesizing control unit 10. “001”, “002”. ． ． ． Is a key for the composite information for editing such as cutting and pasting at the required positions of the image material and the audio material.
Note that two-dimensional 2DCG can be used as the CG data instead of 3DCG. In addition to mov, H.264. Video formats such as H.264, MJEG, and MPEG4 can also be used. These CGs are created by using a software tool such as 3ds Max (registered trademark).

  In addition, the CGDB (stage) 11 is set in advance with the camera position, the angle of view, the lighting position, the number, etc. in order to create a set for photographing a person (including a desk, a chair, etc.) based on a script and a storyboard. , A movie for one title is produced separately from this video / audio synthesizer. After that, any medium such as a CD-ROM, a DVD, and a hard disk, which has been accumulated in the recording medium, can be used, and the CG image is supplied from the recording medium to the computer.

  The CG stage video reproduction unit 12 has a function of extracting and reproducing CG data in accordance with a person who has actually taken a CG video.

  The live-action DB (JPN) 14 stores only acting images by Japanese, the live-action DB (CHN) 15 stores only acting images by Chinese, and the live-action DB (USA) 16 stores only acting images by American. That is, the whole of one movie is recorded as the data of a live-action image of a person acting in a blue background environment, in which only the person image is cut out.

  The live-action person video playback unit 17 has a function of extracting and playing back video data from the live-action DB (JPN) 14 when producing Japanese video content.

  The shooting data reference DB 13 holds temporal information for sharing and synchronizing composite information of the CGDB (stage) 11 and the live-action DB (JPN) 14. That is, the CGDB (stage) 11 and the live-action DB (JPN) 14 are produced independently, but combined information to be linked is required to enable shooting in the same state as the CG stage. That is, to the CG data of the CGDB (stage) 11 and the person image data stored in the live-action image DB 14, the composite information (JPN001 or the like) is added in advance so as to form a set with the CG image data. The combination with and is performed using this combination information.

  The live-action / CG synthesizing section 18 has a function of arranging material images and images in a hierarchical manner to create a complicated shape and to add a special effect. In the present embodiment, the frame of the person image data live-action JPN001 to which the combined information from the photographed person image reproducing unit 17 and the CG data CGJPN001 to which the combined information from the CG stage image reproducing unit 12 is added is used as an input source. It is generated in one image. For example, as shown in FIG. 3B, the real images JPN001 and CGJPN001 are sequentially combined with the combined information JPN001 and JPN002 along the time axis t. As a result, the area of one color component of the video image is replaced with the CG image to generate a composite image.

  The image output unit 19 has a function of connecting the images combined by the live-action / CG composition unit 18 to produce one long image. In the present embodiment, a number of image materials such as the person image data live-action JPN001 and the CG stage image CG data CGJPN001 can be combined and output so as to be arranged as one image, and these combined images can be confirmed.

  It should be noted that in the present embodiment, the foreigner acts in a common set for composite photography under the same conditions such as the camera position, the lighting, and the microphone position for recording the voice of the Japanese performance. Therefore, when synthesizing the photographed image and the CG image, there is an effect that the work of adjusting or correcting the mixture ratio of the images, the position information, and the like becomes unnecessary.

[Sound part]
The sound unit 3 stores a recorded voice DB (JPN) 20 in which voices of Japanese people are accumulated, a recorded voice reproduction unit 23 in which dialogue is reproduced in accordance with a person who is recorded in real time, and SE / BGM for sound synthesis. The SE / BGMDB 24, the SE / BGM reproducing unit 25 read and reproduced from the SE / BGMDB 24, the SE / BGM reproduced by the SE / BGM reproducing unit 26, and the human voice read and reproduced by the recorded voice unit 23 are synthesized. The sound / BGM synthesis unit 26 and the sound output unit 27 capable of outputting and monitoring the sound synthesized by the voice / BGM synthesis unit 26.

  A recorded voice DB (CHN) 21 in which voices by Chinese are stored and a recorded voice DB (USA) 22 in which voices by Americans are stored are shown in FIG. 1, but these are Japanese version movies. Is used when making a Chinese or American movie after it has been produced, not in the process of making a Japanese movie. DB is an abbreviation of DataBase, and the database structure of the recorded voice DB (CHN) 21 and the recorded voice DB (USA) 22 is the same as the database structure and structure of the recorded voice DB (JPN) 20. It's not just a word. These are used when making a Chinese version movie or an American version movie after the Japanese version movie is produced, and are not necessary when making a Japanese version movie.

The SE / BGMDB 24 is composed of wav format music data information in which video sound effects are recorded and accumulated as necessary. For example, FIG. 3 (c) is a diagram schematically showing a data format. SE / BGM data is SE / BGMJPN001, and voice JPN001 is sequentially synthesized from the synthesis information JPN001 as a starting point along the time axis t. Then, the sound is displayed. These synthesizing operations are performed by the synthesizing information from the video / sound synthesizing control unit 10.
The sound data may be stored not only in uncompressed wav but also in compressed MP3 or WMA and decompressed at the time of reproduction.

  The SE / GBMDB 24 is produced in advance by a screenplay and storyboard, and a movie for one title is produced and accumulated in a recording medium by a work different from the function of the video / audio synthesizer. For example, a CD-ROM or a DVD, Any medium such as a hard disk can be used.

  The SE / BGM reproducing unit 25 has a function of extracting and reproducing SE / BGM data. This reproduction function is given a necessary instruction by the video / sound synthesis control unit 10, and the SE / BGM data is transferred to the voice / BGM synthesis unit 26.

The voice / BGM synthesizing unit 26 keeps, for example, take 1 of the voice, superimposes take 2 and take 3 and keeps all takes, copies them to the track, and when the takes have been selected, arranges the joints with a crossfade. It has functions such as going. In the present embodiment, the voice JPN001 of the human voice data to which the synthetic information from the recorded voice reproducing unit 23 is added and the SE / BGM data SE / BGMJPN001 to which the synthetic information from the SE / BGM reproducing unit 12 is added are used as input sources. Combined into one sound. For example, as shown in FIG. 3 (c), the voice JPN001 and SE / BGM001 are sequentially synthesized along the time axis t to keep the whole.
Here, SE / BGM001 is not attached with SE / BGMJP001 because it is a common sound effect regardless of whether it is the Japanese version, the Chinese version, or the American version.
Then, in the sound output unit 27, the sound in which the voice and the sound effect are combined is output separately in stereo, 5.1ch, etc., and the combined output sound can be monitored and confirmed.

It should be noted that the sound unit 3 does not have a function of independently holding temporal information for sharing and synchronizing synthetic information such as the shooting data reference DB 13 in the video unit 2. That is, that is, "JPN001", "JPN002". ． ． ． The code is the composition information. The code of this composite information is a key for cutting, pasting and editing at the required positions of the video material and the audio material.
As a result, it is possible to produce the image and sound by a simple synthesis process without requiring complicated control or operation.

[Video / sound synthesis output section]
Finally, as shown in FIG. 3D, in the video / sound synthesis output unit 28, the synthesized mov format video from the video output unit 19 and the wav format sound from the sound output unit 27 are combined. , The Japanese version of the video content 30 is produced. The video / sound synthesis output unit 28 can synthesize different image data and audio data formats.
Then, this video content can be displayed and monitored for final confirmation.

[Production of video translation contents by video / audio synthesizer]
The function of the video / speech synthesizer 1 for producing the video translation content of the Chinese version (CHN version 31) and the video translation content of the US version (USA version 32) from the video content of the Japanese version (JPN version 30) will be described.
In the present embodiment, the video section 2 selects the live-action DB (CHN) 15 for the Chinese version of the video-translated content (CHN version 31) and the live-action DB (USA) for the US version of the video-translated content (USA version 32). ) 16 is selected and is connected to the photographed data reference DB 13 and the live-action person image reproducing section 17.
Similarly, the sound section 3 selects the recorded audio DB (CHN) 21 for the Chinese version of the video translated content (CHN version 31) and the recorded audio DB (USA) for the US version of the video translated content (USA version 32). 22 is selected and connected to the recorded voice reproducing unit 23.
1 shows the Chinese version of the video translation content (CHN version 31) and the US version of the video translation content (USA version 32), but it is limited to these if other foreign people are available. Instead, for example, Korean version video translation content, French version video translation content, etc. can be produced.

When the production of the foreign version of the video translation content is started, the live-action DB (CHN) 15 is selected for the Chinese version of the video-translated content (CHN version 31), and the live-action DB (for the US version of the video-translated content (USA version 32) is selected. (USA) 16 is selected and connected to the photographed data reference DB 13 and the live-action person image reproduction section 17.
Similarly, the sound section 3 selects the recorded audio DB (CHN) 21 for the Chinese version of the video translated content (CHN version 31) and the recorded audio DB (USA) for the US version of the video translated content (USA version 32). 22 is selected and connected to the recorded voice reproducing unit 23.
As a result, the function of replacing the Japanese in the already-produced video content with a foreign person and producing video-translated content is complete.

  That is, in the case of the Chinese version of the video translated content (CHN version 31), the video section 2 of FIG. 1 is displayed on the CGDB (stage) 11 in which the background video of the computer graphic (CG) is accumulated, and the person who has actually shot the CG video. A CG stage video playback unit 12 that plays back together, a live-action DB (CHN) 15 in which acting videos by the Chinese are accumulated, and a live-action person video playback unit 17 that reads out and plays back a person video from the live-action DB (CHN) 14. A photographic data reference DB 13 that shares data information between the CG DB (stage) 11 and the live-action DB (CHN), a CG image played back by the CG stage video playback unit 12, and a live-action person video playback unit 17 are read and played back. It is composed of a live-action / CG synthesizing section 18 for synthesizing a person video and a video output section 19 for outputting and displaying the video synthesized by the live-action / CG synthesizing section 18. It has been.

  Further, the sound unit 3 includes a recorded voice DB (CHN) 21 in which dialogue voices by Chinese people are accumulated, a recorded voice reproduction unit 23 which reproduces dialogues in accordance with a person who has been recorded, and SE / BGM for sound synthesis. The accumulated SE / BGMDB 24, the SE / BGM reproducing unit 25 read and reproduced from the SE / BGMDB 24, the SE / BGM reproduced by the SE / BGM reproducing unit 26, and the human voice read and reproduced by the recorded voice unit 23 And a sound output unit 27 that can output and monitor the sound synthesized by the voice / BGM synthesis unit 26.

  As explained in the process of the actual recording of the image in Fig. 2, on the same screenplay that the Japanese played, on the composite set (stage) for shooting people based on storyboards, foreigners performed along the same screenplay and actors. A live-action and voice recording of the movie will be held.

  Specifically, the camera position, the angle of view, the illumination position, the microphone position, etc. are set under exactly the same conditions. As a result, the sizes of the images played and photographed by the Japanese people are the same, so that the position synthesis information is unnecessary. Therefore, the CG images and SE / BGM materials produced by the pre-processing process common to the Japanese version of the video content and the foreign version of the video translated content can be used as they are, and the person in the composition set is Japanese. Although it was a word, it is rational to produce foreign version video translation content such as Chinese version and US version by being able to replace the person in the composition set with a foreign language.

  As described above, the viewer recognizes that it is a video produced in his / her country by the video / speech synthesis method that easily produces video translation content in which the person who appears in the Japanese version of the video content is the person in each broadcasting country. By improving liking, empathy and related consumption, it is possible to dramatically improve Japanese movie art.

1 Video / Sound Synthesis Device 2 Video Section 3 Sound Section 10 Video / Sound Synthesis Control Section 11 CG DB (Stage)
12 CG stage image generation unit 13 Shooting data reference DB
14 Live-action DB (JPN)
15 Live-action DB (CHN)
16 Live-action DB (USA)
17 Live-action person video playback unit 18 Live-action / CG composition unit 19 Video output unit 20 Recorded voice DB (JPN)
21 Recorded Voice DB (CHN)
22 Recorded Voice DB (USA)
23 Recorded sound reproduction section 24 SE / BGM DB
25 SE / BGM playback unit 26 Voice / BGM synthesis unit 27 Sound output unit 28 Video / sound synthesis output unit 30 JPN version (video content)
31 CHN version (video translation content)
32 USA version (video translation content)

Claims (2)

A method for synthesizing video and audio, which comprises:
A step of image composition for associating and synthesizing a live-action person image actually taken by a camera and a stage image which is a background of a person produced by computer graphic,
A sound synthesizing step of synthesizing the recorded voice of the photographed person and a sound sound which is a background of the recorded voice in association with each other;
Furthermore, a step of synthesizing the output of the image synthesis step and the output of the sound synthesis step to produce a video content,
Replacing with a live-action portrait image of a person different from the live-action portrait image of the video content;
A video synthesizing step of synthesizing the live-action person of the different person and the video image in association with each other,
Replacing the recorded sound of the video content with the recorded sound of a person, and synthesizing the sound by associating with the sound,
The method according to claim 1, further comprising the step of synthesizing the output of the video synthesizing step of the different person and the output of the sound synthesizing step to produce a video translated content. 2. The video / audio synthesizing method according to claim 1, wherein the synthesizing of the photographed person image and the recorded voice and the synthesization of the photographed person image of the different person and the recorded voice are performed in synchronization.

Images