CN106340291A - Bilingual subtitle making method and system - Google Patents

Abstract

The invention is suitable for the technical field of computers, and provides a bilingual subtitle making method and a bilingual subtitle making system, wherein the method comprises the following steps: the method comprises the steps of receiving an audio and video file input by a user, extracting audio in the audio and video file, dividing the audio into a plurality of audio segments, recording time information of each audio segment, carrying out voice recognition on the plurality of audio segments to generate a subtitle text belonging to a first language, translating the subtitle text belonging to the first language into a subtitle text belonging to a second language, and outputting the subtitle text belonging to the first language and the subtitle text belonging to the second language according to the time information. The method and the device realize the automatic addition of the bilingual subtitles to the non-subtitle audio and video under the condition of no text draft, solve the problems of low efficiency, long time consumption and labor consumption of the bilingual subtitle production in the prior art, and reduce the production cost of the bilingual subtitles.

Description

A method and system for making bilingual subtitles

technical field

The invention belongs to the technical field of computers, and in particular relates to a method and system for making bilingual subtitles.

Background technique

With the development of network video technology, audio programs such as music and radio stations, as well as video programs such as TV dramas, movies, variety shows, and live webcasts have become an indispensable part of people's spare time life. Among them, TV dramas, movies, variety shows, etc. The program is very popular. People not only want to appreciate these original audio and video programs, but also understand the language meaning of these programs well, and they cannot do without the help of bilingual subtitles.

However, in the existing bilingual subtitle production technology, subtitle personnel listen to the audio and record the subtitle text, manually translate, use subtitle production software, and manually add a time axis to obtain the final subtitle file. The existing technology of bilingual subtitle production has the disadvantages of low efficiency, long time consumption, and manpower consumption, and cannot meet the needs of users to generate subtitles for unsubtitled videos at any time.

Contents of the invention

The purpose of the present invention is to provide a method and system for making bilingual subtitles, aiming at solving the problem of low efficiency, time-consuming and manpower-consuming bilingual subtitle production due to the inability to provide an efficient bilingual subtitle production method in the prior art.

On the one hand, the present invention provides a kind of bilingual subtitle production method, described method comprises the following steps:

Receive the audio and video files input by the user, and extract the audio in the audio and video files;

The audio is divided into multiple audio segments, and the time information of each audio segment is recorded;

performing speech recognition on the plurality of audio segments to generate subtitle text belonging to the first language;

Translating the subtitle text belonging to the first language into subtitle text belonging to the second language;

According to the time information, the subtitle text belonging to the first language and the subtitle text belonging to the second language are output.

On the other hand, the present invention provides a kind of bilingual subtitle production system, and described system comprises:

The audio acquisition unit is used to receive the audio and video files input by the user, and extract the audio in the audio and video files;

an audio segmentation unit, configured to divide the audio into multiple audio segments, and record the time information of each audio segment;

A speech recognition unit, configured to perform speech recognition on the plurality of audio segments to generate subtitle text belonging to the first language;

a text translation unit for translating the subtitle text belonging to the first language into subtitle text belonging to the second language; and

The bilingual subtitle generating unit is configured to output the subtitle text belonging to the first language and the subtitle text belonging to the second language according to the time information.

After receiving the audio and video files input by the user, the present invention extracts the audio in the audio and video files, divides the audio into multiple audio segments, records the time information of each audio segment, performs speech recognition on the multiple audio segments, and generates For the subtitle text in one language, translate the subtitle text belonging to the first language into the subtitle text belonging to the second language, and output the subtitle text belonging to the first language and the subtitle text belonging to the second language according to the time information, thereby improving the bilingual subtitle text. The production efficiency of bilingual subtitles is reduced, and bilingual subtitles can be automatically and quickly provided for videos without subtitles.

Description of drawings

Fig. 1 is the implementation flowchart of the bilingual subtitle production method that the embodiment of the present invention provides;

Fig. 2 is the implementation flowchart of dividing the audio into a plurality of audio segments and recording the time information steps of each audio segment in the bilingual subtitle production method provided by Embodiment 2 of the present invention;

Fig. 3 is the implementation flowchart of the step of performing speech recognition on multiple audio segments and generating subtitle text belonging to the first language in the bilingual subtitle production method provided by Embodiment 3 of the present invention;

4 is a flow chart of the steps of translating the subtitle text belonging to the first language into the subtitle text belonging to the second language in the bilingual subtitle production method provided by Embodiment 4 of the present invention;

FIG. 5 is a structural diagram of a bilingual subtitle production system provided by Embodiment 5 of the present invention; and

FIG. 6 is a structural diagram of a bilingual subtitle production system provided by Embodiment 6 of the present invention;

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The specific realization of the present invention is described in detail below in conjunction with specific embodiment:

Embodiment one:

Fig. 1 shows the implementation process of the bilingual subtitle production method provided by Embodiment 1 of the present invention. For the convenience of description, only the parts relevant to the embodiment of the present invention are shown, and the details are as follows:

In step S101, an audio and video file input by a user is received, and audio in the audio and video file is extracted.

The embodiment of the present invention is applicable to audio and video playback devices such as computers and smart phones that support audio and video playback, wherein the audio and video files are audio or video without subtitles, and when the audio and video files are video files, the video files need to be separated Audio stream, save the audio stream as an audio file to facilitate subsequent segmentation and recognition operations.

Preferably, after extracting the audio in the audio and video file, the audio can be preprocessed. The preprocessing method includes audio signal denoising, decibel value setting, etc., to remove noise interference and weaken the background sound in the audio, so that in the audio The pronunciation is clearer, resulting in audio that is more suitable for speech recognition.

In step S102, the audio is divided into multiple audio segments, and the time information of each audio segment is recorded.

In the embodiment of the present invention, the voice pause time interval threshold and the width threshold of the display screen of the audio and video playback device playing the audio are preset, and the audio is segmented according to these two thresholds. The specific segmentation method is as described in Embodiment 2 , which will not be repeated here. Multiple audio segments are obtained after division, wherein the time information of each audio segment includes the start time and end time of the audio segment when the entire audio segment is played, and the start time and end time are used to calculate the duration interval of each audio segment and A timeline for subtitle text.

In step S103, voice recognition is performed on a plurality of audio segments to generate subtitle text belonging to the first language.

In the embodiment of the present invention, according to the voice features corresponding to the audio segment, the subtitle segment corresponding to each audio segment is identified, and then the subtitle text represented by the first language is obtained, where the first language is the original language of the audio segment .

In step S104, the subtitle text belonging to the first language is translated into the subtitle text belonging to the second language.

In the embodiment of the present invention, the language of the other subtitle in the double subtitle is preset, and for the convenience of description, this language is referred to as the second language here. After the subtitle text in the first language is obtained, the subtitle text in the first language can be translated into the subtitle text in the second language through a translation program or system capable of inter-translation among multiple languages.

In step S105, according to the time information, the subtitle text belonging to the first language and the subtitle text belonging to the second language are output.

In the embodiment of the present invention, the time information refers to the time information of each audio segment, which may include the corresponding start time and end time of each audio segment in the complete audio playback process. According to the start time and end time, respectively belong to the first The subtitle text of one language and the subtitle text of the second language establish a time axis, and the time axis is used for calibration between subtitle display and video playback, according to the time of the subtitle text of the first language and the subtitle text of the second language The corresponding relationship between axes, synchronously output subtitle content in two languages, and finally get bilingual subtitles, so as to realize the automatic generation or production of bilingual subtitles, and improve the production efficiency of bilingual subtitles.

Embodiment two:

Figure 2 shows the implementation process of dividing the audio into multiple audio segments and recording the time information of each audio segment in the bilingual subtitle production method provided by Embodiment 2 of the present invention. The relevant parts of the example are described in detail below:

In step S201, the audio is preliminarily segmented according to the speech pause of the audio.

In the embodiment of the present invention, a speech pause time interval threshold for segmenting audio is preset. Since people usually have a short pause between two sentences when speaking, it is possible to detect the pause or silence in the audio, and calculate the time interval of the pause or silence. When the time interval is greater than the preset voice Split the audio when the pause interval threshold is reached. In this way, the operation of segmenting the audio according to the pause of speech can be realized, which can effectively improve the accuracy of the subsequent speech recognition of the audio segment.

In step S202, according to the width of the display screen, the pre-segmented audio is divided and merged twice to obtain multiple audio segments, and the time information of each audio end is recorded.

In the embodiment of the present invention, after the audio is initially divided, audio segments with different duration intervals are obtained. Some audio segments with long duration intervals exceed the width of the display screen of the audio and video display device playing the video, and cannot be completely displayed. Some audio segments with short duration intervals are too short to be viewed individually. Therefore, the width threshold of the display screen of the audio and video playback device playing the audio can be preset, and the longest audio time interval threshold and the shortest audio time interval threshold can be set according to the width threshold of the display screen. Traverse the audio segments obtained after the initial segmentation, and perform secondary segmentation on the audio segments whose duration interval is greater than the threshold of the longest audio time interval. The average split method can be used. For an audio segment whose duration interval is smaller than the shortest audio time interval threshold after preliminary segmentation, query the duration interval of adjacent audio segments of the audio segment, and merge the audio segment with the adjacent audio segment with a shorter duration interval. Finally, multiple audio segments are obtained through the three-step operation of primary segmentation, secondary segmentation, and merging, which can effectively improve the accuracy of subsequent speech recognition of audio segments.

The time information of the audio segment includes the corresponding start time and end time of the audio segment during the entire audio playback process. This time information is used to calculate the duration interval of the audio segment in the segmentation operation and for the production of the subsequent subtitle text time axis.

Preferably, in order to avoid segmentation errors and reduce errors in the segmentation process, in the preliminary segmentation and secondary segmentation process, only the start time and end time of the audio segment can be marked, and the actual segmentation behavior is not performed. Like this, After the first marking, the marking points can be further adjusted continuously, so that the divided audio segment can be adjusted multiple times again, and the actual division operation can be performed after the adjustment is completed.

Embodiment three:

Fig. 3 shows the implementation process of performing voice recognition on multiple audio segments and generating subtitle texts belonging to the first language in the bilingual subtitle production method provided by Embodiment 3 of the present invention. The relevant part of the embodiment is described in detail as follows:

In step S301, the audio segment to be recognized is matched in the pre-built high-frequency speech recognition library.

In the embodiment of the present invention, the speech features of the audio segment to be recognized are extracted, the speech features are input into the pre-built high-frequency speech recognition library, and the speech features in the high-frequency speech recognition library are matched to obtain the speech recognition result. The pre-built high-frequency speech recognition library collects common speech and the corresponding recognition results of the speech, thereby improving the accuracy and efficiency of speech recognition.

In step S302, when the audio segment to be identified is successfully matched, the subtitle text in the first language corresponding to the audio segment to be identified is acquired.

In the embodiment of the present invention, when the audio segment to be recognized is successfully matched, the speech recognition result is recorded in the first language, and the subtitle text belonging to the first language is obtained. When the audio segment to be recognized is unsuccessfully matched, the speech segment to be recognized can be input into a preset voice recognition system for recognition. Preferably, the preset voice recognition system can be the voice recognition system of Youtong Technology and the voice recognition system of HKUST Xunfei, so that the accuracy of the recognition result can be effectively improved.

Embodiment four:

Fig. 4 shows the implementation flow of the step of translating the subtitle text belonging to the first language into the subtitle text belonging to the second language in the bilingual subtitle production method provided by Embodiment 4 of the present invention. The parts related to the embodiments of the invention are described in detail as follows:

In step S401, the subtitle text belonging to the first language is divided into multiple text segments.

In step S402, the text segment to be translated is matched in the pre-built high-frequency word translation database.

In the embodiment of the present invention, the subtitle text belonging to the first language is divided to obtain text segments that are easy to translate, and the text segments to be translated are input into a pre-built high-frequency word translation database to obtain translation results. The pre-built high-frequency word translation database collects common words or phrases, as well as the corresponding translation results of these words and phrases, which can effectively improve the accuracy and efficiency of text translation.

In step S403, when the text segment to be translated is successfully matched, the subtitle text corresponding to the text segment to be translated and belonging to the second language is acquired.

In the embodiment of the present invention, when the text segment to be translated is successfully matched, the translation result is recorded in the second language, and the subtitle text belonging to the second language is obtained. When the text segment to be translated is not successfully matched, the text to be translated can be input into a preset translation system. Preferably, the preset translation system can be google online translation system, Youdao online translation system, etc., which can effectively improve the accuracy of translation results.

Those of ordinary skill in the art can understand that all or part of the steps in the method of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage Media such as ROM/RAM, magnetic disk, optical disk, etc.

Embodiment five:

FIG. 5 shows the structure of the bilingual subtitle production system provided by the fifth embodiment of the present invention. For the convenience of description, only the parts related to the embodiment of the present invention are shown.

In an embodiment of the present invention, the bilingual subtitle production system includes an audio acquisition unit 51, an audio segmentation unit 52, an audio recognition unit 53, a text translation unit 54, and a bilingual subtitle generation unit 55, wherein:

The audio acquisition unit 51 is configured to receive audio and video files input by the user, and extract audio from the audio and video files.

The audio segmentation unit 52 is configured to divide the audio into multiple audio segments, and record the time information of each audio segment.

The voice recognition unit 53 is configured to perform voice recognition on a plurality of audio segments to generate subtitle text belonging to the first language.

The text translation unit 54 is configured to translate the subtitle text belonging to the first language into the subtitle text belonging to the second language.

The bilingual subtitle generation unit 55 is configured to output the subtitle text belonging to the first language and the subtitle text belonging to the second language according to the time information.

In the embodiment of the present invention, each unit of the bilingual subtitle production system can be realized by corresponding hardware or software units, and each unit can be an independent software and hardware unit, or can be integrated into a software and hardware unit of a computer device or system. This is not intended to limit the invention. For the specific implementation manner of each unit of the bilingual subtitle production system, reference may be made to the description of the corresponding steps in Embodiment 1, which will not be repeated here.

Embodiment six:

Figure 6 shows the structure of the bilingual subtitle production system provided by Embodiment 6 of the present invention. For ease of description, only the parts related to the embodiment of the present invention are shown. The bilingual subtitle production system includes:

Audio acquisition unit 61, for receiving the audio-video file input by the user, extracting the audio in the audio-video file;

Audio segmentation unit 62, for audio is divided into a plurality of audio segments, record the time information of each audio segment;

Speech recognition unit 63, for carrying out speech recognition to a plurality of audio segments, generating the subtitle text belonging to the first language;

A text translation unit 64, configured to translate the subtitle text belonging to the first language into the subtitle text belonging to the second language; and

The bilingual subtitle generating unit 65 is configured to output subtitle text belonging to the first language and subtitle text belonging to the second language according to the time information.

In the embodiment of the present invention, preferably, the audio segmentation unit 62 includes:

A preliminary processing unit 621, configured to perform preliminary segmentation on the audio according to the speech pause of the audio; and

The secondary processing unit 622 is configured to perform secondary segmentation and merging on the pre-segmented audio according to the width of the display screen.

Preferably, the voice recognition unit 63 includes:

Voice matching unit 631, for matching the audio to be recognized in the pre-built high-frequency voice library; and

The first text generating unit 632 is configured to obtain subtitle text corresponding to the audio segment to be recognized and belonging to the first language when the speech to be recognized is successfully matched.

Preferably, the text translation unit 64 includes:

A text division unit 641, configured to divide the subtitle text belonging to the first language into a plurality of text segments;

A text matching unit 642, configured to match the text segment to be translated in the pre-built high-frequency word translation database; and

The second text generation unit 643 is configured to obtain subtitle text in the second language corresponding to the text segment to be translated when the text segment to be translated is successfully matched.

Preferably, the bilingual subtitle generation unit 65 includes:

A time axis creation unit 651, configured to construct a first time axis of the subtitle text belonging to the first language and a second time axis of the subtitle text belonging to the second language according to the time information; and

The bilingual subtitle generating subunit 652 is configured to synchronously output the subtitle text belonging to the first language and the subtitle text belonging to the second language according to the corresponding relationship between the first time axis and the second time axis, so as to obtain bilingual subtitles.

In the embodiment of the present invention, according to the start time and the end time, a time axis is respectively established for the subtitle file belonging to the first language and the subtitle text belonging to the second language, and then according to the subtitle text belonging to the first language and the subtitle text belonging to the second language The corresponding relationship between the time axis of the alphabetic text, synchronously output subtitle content in two languages to obtain bilingual subtitles, thereby improving the generation efficiency of bilingual subtitles, reducing the production cost of bilingual subtitles, and realizing automatic and fast provision of subtitle-free videos Bilingual subtitles.

In the embodiment of the present invention, each unit of the bilingual subtitle system device can be realized by a corresponding hardware or software unit, and each unit can be an independent software and hardware unit, or can be integrated into a software and hardware unit, which is not intended to limit this invention. For the specific implementation manner of each unit of the bilingual subtitle production system, reference may be made to the description of the corresponding steps in the foregoing embodiments, and details are not repeated here.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (10)

1. a kind of bilingual subtitles preparation method is it is characterised in that methods described comprises the steps:

The audio-video document of receiving user's input, extracts the audio frequency in described audio-video document；

Described audio segmentation is multiple audio sections, records the temporal information of each audio section；

The plurality of audio section is carried out with speech recognition, generates the captioned test belonging to the first languages；

The described captioned test belonging to the first languages is translated as belonging to the captioned test of the second languages；

According to described temporal information, belong to the captioned test of the first languages described in output and belong to the captioned test of the second languages.

2. the method for claim 1, it is characterised in that being multiple audio sections by described audio segmentation, records each sound The step of the temporal information of frequency range, comprising:

According to the speech pause of described audio frequency, primary segmentation is carried out to described audio frequency；

According to the width of display screen, secondary splitting and merging are carried out to the described audio frequency after primary segmentation, described many to obtain Individual audio section, and record the temporal information of each audio section.

3. it is characterised in that carrying out speech recognition to the plurality of audio section, generation belongs to the method for claim 1 The step of the captioned test of the first languages, comprising:

The High frequency speech identification storehouse building in advance is mated to audio section to be identified；

When the match is successful to described audio section to be identified, obtain that described audio section to be identified is corresponding belongs to the first languages Captioned test.

4. the method for claim 1 is it is characterised in that be translated as belonging to the described captioned test belonging to the first languages The step of the captioned test of the second languages, comprising:

The described captioned test belonging to the first languages is divided into multiple text chunks；

The high frequency words translation storehouse building in advance is mated to text chunk to be translated；

When the match is successful to described text chunk to be translated, obtain that described text chunk to be translated is corresponding belongs to the second languages Captioned test.

5. the method for claim 1 is it is characterised in that according to described temporal information, belong to the first languages described in output Captioned test and belong to the second languages captioned test step, comprising:

Belong to the very first time axle of the captioned test of the first languages according to described temporal information builds, belong to the second languages Second time shaft of captioned test；

According to the corresponding relation of described very first time axle and described second time shaft, belong to the word of the first languages described in synchronism output Curtain text and the captioned test belonging to the second languages, to obtain bilingual subtitles.

6. a kind of bilingual subtitles manufacturing system is it is characterised in that described system includes:

Audio frequency acquiring unit, for the audio-video document of receiving user's input, extracts the audio frequency in described audio-video document；

Audio segmentation unit, for being multiple audio sections by described audio segmentation, records the temporal information of each audio section；

Voice recognition unit, for the plurality of audio section is carried out with speech recognition, generates the captioned test belonging to the first languages；

Text translation unit, for being translated as belonging to the captions literary composition of the second languages by the described captioned test belonging to the first languages This；And

Bilingual subtitles signal generating unit, for according to described temporal information, belonging to captioned test and the genus of the first languages described in output Captioned test in the second languages.

7. system as claimed in claim 6 is it is characterised in that described audio segmentation unit includes:

Preliminary treatment unit, for the speech pause according to audio frequency, carries out primary segmentation to described audio frequency；And

After-treatment unit, for the width according to display screen, the described audio frequency after primary segmentation is carried out secondary splitting and Merge, to obtain the plurality of audio section, and record the temporal information of each audio section.

8. system as claimed in claim 6 is it is characterised in that described voice recognition unit includes:

Voice match unit, for mating to audio section to be identified in the High frequency speech identification storehouse building in advance；With And

First text generation unit, for, when successful to described Audio Matching to be identified, obtaining described audio frequency to be identified The corresponding captioned test belonging to the first languages of section.

9. system as claimed in claim 6 is it is characterised in that described text translation unit, comprising:

Text division unit, for being divided into multiple text chunks by the described captioned test belonging to the first languages；

Text matches unit, for mating to text chunk to be translated in the high frequency words translation storehouse building in advance；And

Second text generation unit, for when the match is successful to described text chunk to be translated, obtaining described literary composition to be translated This section of corresponding captioned test belonging to the second languages.

10. system as claimed in claim 6 is it is characterised in that described bilingual subtitles signal generating unit includes:

Time shaft production unit, for according to described temporal information build described in belong to the first languages captioned test first when Countershaft, belong to the second languages captioned test the second time shaft；And

Bilingual subtitles generate subelement, for the corresponding relation according to described very first time axle and described second time shaft, synchronous The captioned test belonging to the first languages described in output and the captioned test belonging to the second languages, to obtain bilingual subtitles.