CN108363704A - A kind of neural network machine translation corpus expansion method based on statistics phrase table - Google Patents

Abstract

The invention discloses a neural network machine translation corpus extension method based on a statistical phrase table, which belongs to the technical field of machine translation. The present invention proposes a machine translation corpus expansion method based on a statistical phrase table for the neural network machine translation technology, which can effectively expand the corpus scale on the basis of the original training set of machine translation; the method mainly includes: training set expansion stage and model training Stage; Stage 1 uses statistical machine learning methods to learn phrase tables from the original training set and fuses it with the original training set according to certain filtering rules to form a new expanded training set. Stage 2 trains the neural machine translation model. The expanded training set is pre-trained, and then the original training set is used for training and tuning to obtain the final model; the experimental results show that compared with the machine translation model that does not use the corpus expansion method, the BLEU evaluation index of the present invention is significantly improved.

Description

A Neural Network Machine Translation Corpus Expansion Method Based on Statistical Phrase Table

technical field

The invention relates to a neural network machine translation corpus extension method based on a statistical phrase table, and belongs to the technical fields of computer application and machine translation.

Background technique

Machine translation is a technology that uses computers to automatically translate one language (source language) into another language (target language).

With the development of artificial neural network and deep learning technology, the neural network machine translation technology based on deep learning technology (hereinafter referred to as neural machine translation) has made important achievements in recent years. Neural machine translation has the advantages of less linguistic knowledge and manual intervention, less space for model storage, and smooth and natural translation output. Neural machine translation is generally considered the best choice for bilingual resource-rich translation tasks. At present, neural machine translation has received extensive attention and recognition in the field of machine translation, and has been put into commercial use.

The data for training the neural network is mainly bilingual parallel sentence pairs. Usually, the neural network models used in neural machine translation have large-scale free parameters. In theory, such models require large-scale bilingual parallel corpora to train them. Experience has shown that a neural machine translation model containing tens of millions of free parameters usually requires at least one million sentence-pair level of data for training to achieve ideal results. For some languages where bilingual parallel resources are relatively scarce, it is difficult to obtain satisfactory results by applying neural networks for translation.

In addition, the training of neural machine translation is usually carried out in units of one or a group (multiple) complete sentence pairs. When the corpus resources are scarce, the ability to learn some phrases with low frequency included in the sentence pairs is limited. Especially when translating these phrases individually.

Contents of the invention

Aiming at the model training problem of neural machine translation for languages with scarce resources, the present invention proposes a corpus expansion method for neural network machine translation based on statistical phrase tables, which can effectively expand the training data of neural machine translation models and alleviate the impact of scarce language resources on model training. adverse effects.

The present invention includes: a training set expansion stage and a model training stage;

Among them, A) the operation of the training set expansion stage is as follows: the phrase table with probability score is learned from the original training set by statistical machine learning method, and the learned phrase table is filtered according to the rules, and the filtered phrase table is extracted Form a new bilingual parallel phrase pair data set, splice the newly extracted data set with the original training set to obtain new bilingual parallel pseudo-data, and realize the expansion of the training set;

B) The operation of the model training stage is divided into two steps. Step one is pre-training, that is, the bilingual parallel dummy data obtained in stage A) is used to pre-train the model, and the pre-trained model b ₁ is obtained after training; step two uses the original The training set retrains the model b ₂ , the purpose is to tune the model and alleviate the influence of the noise introduced in the fake data on the model;

For realizing above-mentioned purpose and technology, the technical scheme that the present invention adopts is as follows:

First, the relevant definitions are made, as follows:

Definition 1: Source language, that is, in machine translation, the language of the content to be translated during translation, for example, in machine translation from Chinese to English, Chinese is the source language;

Definition 2: Source language data, that is, data belonging to the source language. If the source language data is a natural language sentence, the data belonging to the source language is called a source language sentence. For example, in machine translation from Chinese to English, the input Chinese sentences are the source language data, which can also be called source language sentences;

A collection of source language data is called a source language dataset;

Definition 3: Target language, that is, in machine translation, the language to which the translated content belongs, for example, in machine translation from Chinese to English, English is the target language;

Definition 4: Target language data, that is, data belonging to the target language. If the target language data is a natural language sentence, the data belonging to the target language is called the target language sentence. For example, in machine translation from Chinese to English, the output English sentences are the target language data, and can also be called target language sentences;

A collection of target language data is called a target language dataset;

Definition 5: Training set, specifically refers to the training set of the statistical machine translation model, that is, the data set used to train the statistical machine translation model, denoted as T;

Definition 6: The original training set, that is, the training set before expansion;

Definition 7: Word alignment information, referred to as word alignment, is the alignment relationship between source language words and target language words in the training set T, denoted as α;

Among them, if in the training set T, there is an alignment relationship between the jth word in the source language and the ith word in the target language, it is recorded as (j,i);

Definition 8, a phrase, a linguistic unit consisting of one or more words;

Phrases in the source language are called source language phrases, denoted as f, and phrases in the target language are called target language phrases, denoted as e;

Definition 9, translation phrase pair, a phrase pair consisting of a source language phrase and an aligned target language phrase, such as "('The Great Wall', 'The Great Wall')";

Definition 10, the forward phrase translation probability, that is, given the source language phrase f, the conditional probability of translation to the target language phrase e, denoted as

Definition 11, reverse phrase translation probability, which is the conditional probability of translating back to the source language phrase f given the target language phrase e. recorded as

Definition 12, two-way phrase translation probability, forward phrase translation probability and reverse phrase translation probability are collectively called two-way phrase translation probability;

Definition 13, the forward lexicalized phrase translation probability, given the source language phrase f, the lexicalized translation probability translated to the target language phrase e, denoted as lex(e|f);

Definition 14, the reverse lexicalized phrase translation probability, given the target language phrase e, the lexicalized translation probability of the translation back to the source language phrase f, denoted as lex(f|e);

Definition 15, two-way lexicalization phrase translation probability, forward lexicalization translation probability and reverse lexicalization translation probability are collectively called two-way lexicalization translation probability;

Definition 16. The phrase table, also called the phrase translation table, is composed of multiple sets of translated phrase pairs, and each set of translated phrase pairs is attached with a bidirectional phrase translation probability and a bidirectional lexicalized translation probability;

Definition 17, filtering rules, that is, the rules for filtering phrase tables, artificially formulated rules for filtering phrase tables according to the source language phrases, target language phrases, bidirectional phrase translation probability, and bidirectional lexicalized phrase translation probability information contained in the phrase table ;

The training set expansion phase includes the following steps:

In step A1, according to definition 1, definition 2, definition 3, definition 4 and definition 5, the original training set is preprocessed to obtain the preprocessed original training set T _f ;

Among them, the specific process of preprocessing the original training set varies with different source languages and target languages. The purpose is to standardize the training set and obtain the preprocessed original training set T _f ;

Step A2, based on the preprocessed original training set T _f obtained in step A1, and learn word alignment information according to Definition 7 and Definition 8, this process is usually implemented by using an open source word alignment toolkit, and the preprocessed training set T f obtained in step A1 The processed original training set is used as input, and the word alignment information α of the training set is obtained through the training of the training word alignment tool;

Step A3, according to Definition 6, Definition 7, Definition 8, Definition 9, Definition 10, Definition 11, Definition 12, Definition 13, Definition 14, Definition 15 and Definition 16, combined with the preprocessed original training set obtained in Step A1 T _f and the word alignment information α of the training set obtained in step A2, extract translation phrase pairs, and estimate the probability of translation phrase pairs, and obtain the bidirectional phrase translation probability and bidirectional lexicalization translation probability of each translation phrase pair, combined with translation phrases pair and translation probability to get a phrase table, each record of the phrase table is composed of translation phrase pair, word alignment information, two-way phrase translation probability and two-way lexicalized translation probability;

Step A4, according to Definition 9, Definition 12, Definition 15, Definition 16, and Definition 17, use manually defined filter rules to filter the phrase table obtained in Step A3, filter out translation phrase pairs with low probability, and obtain the filtered Phrase table, denoted as P _new ;

Step A5, according to Definition 5 and Definition 16, splicing the translated phrase pairs in the filtered phrase table P _new obtained in Step A4 with the preprocessed original training set T _f obtained in Step A1 to obtain a new training set T _new ;

From step A1 to step A5, the training set expansion stage of the method is completed;

The model training phase includes the following steps:

Step B1, using the new training set T _new obtained in step A5 to pre-train the model to obtain model _b1 ;

Step B2, using the preprocessed original training set T _f obtained in step A1, to train the model b 1 obtained in step _B1 again to obtain a newly trained model b ₂ ;

So far, from step B1 to step B2, the model training phase of the method is completed;

So far, from step A1 to step A5 and step B1 to step B2, a method for expanding the corpus of neural network machine translation based on the statistical phrase table has been completed.

Beneficial effect

A kind of neural network machine translation corpus expansion method based on statistical phrase table of the present invention, compared with existing machine translation training set using method, has following beneficial effect:

1. The present invention has designed a neural network machine translation corpus extension method based on a statistical phrase table. This method can effectively expand the original training set without requiring additional bilingual or monolingual data, and alleviate resource-scarce language training. The adverse effect of small set size on the training of neural machine translation models.

2. In the case of the same training set, development set and test set data, compared with the neural machine translation model training method not using the present invention, the BLEU evaluation index of the present invention is significantly improved.

Description of drawings

FIG. 1 is a flow chart of a method and embodiment of a neural network machine translation corpus expansion method based on a statistical phrase table in the present invention.

Detailed ways

The method of the present invention will be described in detail below in conjunction with the accompanying drawings and embodiments. During the description, the two main stages included in the present invention: 1) the training set expansion stage and 2) the model training stage will be described separately.

Example 1

This embodiment describes the flow of the method of the present invention and its specific examples.

FIG. 1 is a statistical phrase table-based neural network machine translation corpus expansion method according to the present invention and a flow chart in this embodiment.

It can be seen from Fig. 1 that the present invention includes two stages 1) the training set expansion stage and 2) the operation flow of the model training stage.

Take Uyghur-to-Chinese translation as an example, where Uyghur is the source language and Chinese is the target language.

1) Training set expansion stage:

Step 1. Preprocess the original training set according to Definition 1, Definition 2, Definition 3, Definition 4, and Definition 5. The specific process of preprocessing varies with different source languages and target languages. The purpose is to standardize the training set. Among them, the preprocessing process of the source language Uyghur and the target language Chinese data are: first perform word-piece segmentation, and then perform word segmentation (tokenization) to obtain the original training set after preprocessing T _f ;

Step 2, according to definition 6 and definition 7, learn word alignment, in the present embodiment, this process utilizes the open source word alignment toolkit GIZA++ to realize, the original training set after preprocessing that obtains in step 1 is used as input, after training The training of the word alignment tool GIZA++ obtains the word alignment information α of the training set;

Step 3, according to Definition 6, Definition 7, Definition 8, Definition 9, Definition 10, Definition 11, Definition 12, Definition 13, Definition 14, Definition 15 and Definition 16, combined with the preprocessed original training set obtained in Step 1 T _f and the word alignment information α of the training set obtained in step 2 extract translation phrases and estimate the probability of translation phrases. In this embodiment, the above functions are realized by using the train-model.perl script in the Moses open source tool. Get the phrase table P, each record of the phrase table consists of translation phrase pairs, word alignment information, bidirectional phrase translation probability and bidirectional lexicalization translation probability;

Step 4, according to Definition 9, Definition 12, Definition 15, Definition 16, and Definition 17, use manually defined filtering rules to filter the phrase list obtained in Step 3. The manually defined rules are as follows:

Keep the translation phrase pair if and only if the probability of the translation phrase pair and And lex(e|f)≥0.025, and lex(f|e)≥0.025;

Filter out the translation phrase pairs with low probability to obtain the filtered new phrase table P _new ;

Step 5, according to Definition 5 and Definition 16, splicing the translated phrase pair part of the filtered new phrase table P _new obtained in Step 4 with the preprocessed original training set T _f obtained in Step 1 to obtain a new training set T _new ;

2) The steps in the model training phase are as follows:

Step 6, perform model pre-training. In this embodiment, the open source neural machine translation model tesnor2tensor is used, and the new training set T _new obtained in step 5 is used to pre-train the model to obtain model _b1 ;

Step 7, use the preprocessed original training set T _f obtained in step 1 to train the model b ₁ obtained in step 6 again to obtain a newly trained model b ₂ ;

So far, from step 1 to step 7, a corpus extension method for neural network machine translation based on statistical phrase table has been completed.

Example 2

The training set in the Uyghur-Chinese news translation task provided by CWMT2017 is randomly split into training set, development set and test set 1. In addition, the development set data of the Uyghur-Chinese news translation evaluation task provided by CWMT2017 is used as a test Set 2, the experimental results show that, in the case that the original training set, development set, test set data and neural machine translation model are the same, the present invention adopts BLEU based on Chinese characters compared with the neural machine translation model training method of the present invention. As the evaluation index, the following experimental results can be obtained.

Table 1 uses the comparison of BLEU values before and after the training set expansion method proposed by the present invention

The experimental results in Table 1 show that: in the case of the same training set, development set and test set data, the BLEU evaluation index is significantly improved by using the method of the present invention compared with the neural machine translation model training method not using the present invention.

The above description is only a preferred embodiment of the present invention, and the present invention should not be limited to the content disclosed in this embodiment and the accompanying drawings. All equivalents or modifications accomplished without departing from the disclosed spirit of the present invention fall within the protection scope of the present invention.

Claims (4)

1. A neural network machine translation corpus expansion method based on a statistical phrase table, characterized in that: comprising: a training set expansion stage and a model training stage; Among them, A) the operation of the training set expansion stage is as follows: the phrase table with probability score is learned from the original training set by statistical machine learning method, and the learned phrase table is filtered according to the rules, and the filtered phrase table is extracted Form a new bilingual parallel phrase pair data set, splice the newly extracted data set with the original training set to obtain new bilingual parallel pseudo-data, and realize the expansion of the training set; B) The operation of the model training stage is divided into two steps. Step one is pre-training, that is, the bilingual parallel dummy data obtained in stage A) is used to pre-train the model, and the pre-trained model b ₁ is obtained after training; step two uses the original The training set retrains the model b ₂ for the purpose of tuning the model and alleviating the influence of the noise introduced in the fake data on the model. 2. a kind of neural network machine translation corpus expansion method based on statistical phrase table according to claim 1, is characterized in that: for realizing above-mentioned purpose and technology, adopt following technical scheme: First, the relevant definitions are made, as follows: Definition 1: Source language, that is, in machine translation, the language of the content to be translated during translation, for example, in machine translation from Chinese to English, Chinese is the source language; Definition 2: Source language data, that is, data belonging to the source language. If the source language data is a natural language sentence, the data belonging to the source language is called a source language sentence. For example, in machine translation from Chinese to English, the input Chinese sentences are the source language data, which can also be called source language sentences; A collection of source language data is called a source language dataset; Definition 3: Target language, that is, in machine translation, the language to which the translated content belongs, for example, in machine translation from Chinese to English, English is the target language; Definition 4: Target language data, that is, data belonging to the target language. If the target language data is a natural language sentence, the data belonging to the target language is called the target language sentence. For example, in machine translation from Chinese to English, the output English sentences are the target language data, and can also be called target language sentences; A collection of target language data is called a target language dataset; Definition 5: Training set, specifically refers to the training set of the statistical machine translation model, that is, the data set used to train the statistical machine translation model, denoted as T; Definition 6: The original training set, that is, the training set before expansion; Definition 7: Word alignment information, referred to as word alignment, is the alignment relationship between source language words and target language words in the training set T, denoted as α; Among them, if in the training set T, there is an alignment relationship between the jth word in the source language and the ith word in the target language, it is recorded as (j,i); Definition 8, a phrase, a linguistic unit consisting of one or more words; Phrases in the source language are called source language phrases, denoted as f, and phrases in the target language are called target language phrases, denoted as e; Definition 9, translation phrase pair, a phrase pair consisting of a source language phrase and an aligned target language phrase, such as "('The Great Wall', 'The Great Wall')"; Definition 10, the forward phrase translation probability, that is, given the source language phrase f, the conditional probability of translation to the target language phrase e, denoted as Definition 11, the reverse phrase translation probability, that is, given the target language phrase e, the conditional probability of translating back to the source language phrase f, denoted as Definition 12, two-way phrase translation probability, forward phrase translation probability and reverse phrase translation probability are collectively called two-way phrase translation probability; Definition 13, the forward lexicalized phrase translation probability, given the source language phrase f, the lexicalized translation probability translated to the target language phrase e, denoted as lex(e|f); Definition 14, the reverse lexicalized phrase translation probability, given the target language phrase e, the lexicalized translation probability of the translation back to the source language phrase f, denoted as lex(f|e); Definition 15, two-way lexicalization phrase translation probability, forward lexicalization translation probability and reverse lexicalization translation probability are collectively called two-way lexicalization translation probability; Definition 16. The phrase table, also called the phrase translation table, is composed of multiple sets of translated phrase pairs, and each set of translated phrase pairs is attached with a bidirectional phrase translation probability and a bidirectional lexicalized translation probability; Definition 17, filtering rules, that is, the rules for filtering phrase tables, artificially formulated rules for filtering phrase tables according to the source language phrases, target language phrases, bidirectional phrase translation probability, and bidirectional lexicalized phrase translation probability information contained in the phrase table ; The training set expansion phase includes the following steps: In step A1, according to definition 1, definition 2, definition 3, definition 4 and definition 5, the original training set is preprocessed to obtain the preprocessed original training set T _f ; Step A2, based on the preprocessed original training set T _f obtained in step A1, and learn word alignment information according to Definition 7 and Definition 8, this process is usually implemented by using an open source word alignment toolkit, and the preprocessed training set T f obtained in step A1 The processed original training set is used as input, and the word alignment information α of the training set is obtained through the training of the training word alignment tool; Step A3, according to Definition 6, Definition 7, Definition 8, Definition 9, Definition 10, Definition 11, Definition 12, Definition 13, Definition 14, Definition 15 and Definition 16, combined with the preprocessed original training set obtained in Step A1 T _f and the word alignment information α of the training set obtained in step A2, extract translation phrase pairs, and estimate the probability of translation phrase pairs, and obtain the bidirectional phrase translation probability and bidirectional lexicalization translation probability of each translation phrase pair, combined with translation phrases pair and translation probability to get a phrase table, each record of the phrase table is composed of translation phrase pair, word alignment information, two-way phrase translation probability and two-way lexicalized translation probability; Step A4, according to Definition 9, Definition 12, Definition 15, Definition 16, and Definition 17, use manually defined filter rules to filter the phrase table obtained in Step A3, filter out translation phrase pairs with low probability, and obtain the filtered Phrase table, denoted as P _new ; Step A5, according to Definition 5 and Definition 16, splicing the translated phrase pairs in the filtered phrase table P _new obtained in Step A4 with the preprocessed original training set T _f obtained in Step A1 to obtain a new training set T _new . 3. a kind of neural network machine translation corpus expansion method based on statistical phrase table according to claim 1, is characterized in that: model training stage, comprises the steps: Step B1, using the new training set T _new obtained in step A5 to pre-train the model to obtain model _b1 ; In step B2, use the preprocessed original training set T _f obtained in step A1 to train the model b 1 obtained in step _B1 again to obtain a newly trained model b ₂ . 4. a kind of neural network machine translation corpus expansion method based on statistical phrase table according to claim 1, is characterized in that: in step A1, wherein, the specific process that the original training set is carried out preprocessing varies from source language and target The purpose is to normalize the training set and obtain the preprocessed original training set T _f .