# Neural Machine Translation Model

## Background Introduction

Neural Machine Translation (NMT) is a simple new architecture for getting machines to learn to translate. Traditional machine translation methods are mainly based on phrase-based statistical translation approaches that use separately engineered subcomponents rules or statistical models. NMT models use deep learning and representation learning. This example describes how to construct an end-to-end neural machine translation (NMT) model using the recurrent neural network (RNN) in PaddlePaddle.

## Model Overview

RNN-based neural machine translation follows the encoder-decoder architecture. A common choice for the encoder and decoder is the recurrent neural network (RNN), used by most NMT models. Below is an example diagram of a general approach for NMT.

Figure 1. Encoder - Decoder frame p> The input and output of the neural machine translation model can be any of character, word or phrase. This example illustrates the word-based NMT. - **Encoder**: Encodes the source language sentence into a vector as input to the decoder. The original input of the decoder is the id sequence $w = {w_1, w_2, ..., w_T}$ of the word, expressed in the one-hot code. In order to reduce the input dimension, and to establish the semantic association between words, the model is a word that is expressed by hot independent code. Word embedding is a word vector. For more information about word vector, please refer to PaddleBook [word vector] (https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/README.cn.md) chapter. Finally, the RNN unit processes the input word by word to get the encoding vector of the complete sentence. - **Decoder**: Accepts the input of the encoder, decoding the target language sequence $u = {u_1, u_2, ..., u_ {T '}}$ one by one. For each time step, the RNN unit outputs a hidden vector. Then the conditional probability of the next target word is calculated by Softmax normalization, i.e. $P (u_i | w, u_1, u_2, ..., u_ {t- 1})$. Thus, given the input $w$, the corresponding translation result is $u$ $$P(u_1,u_2,...,u_{T'} | w) = \prod_{t=1}^{t={T'}}p(u_t|w, u_1, u_2, u_{t-1})$$ In Chinese to English translation, for example, the source language is Chinese, and the target language is English. The following is a sentence after the source language word segmentation. wzxhzdk:0 Corresponding target language English translation results for: wzxhzdk:1 In the preprocessing step, we prepare the parallel corpus data of the source language and the target language. Then we construct the dictionaries of the source language and the target language respectively. In the training stage, we use the pairwise parallel corpus training model. In the model test stage, the model automatically generates the corresponding English translation, and then it evaluates the resulting results with standard translations. For the evaluation metric, BLEU is most commonly used. ### RNN unit The original structure of the RNN uses a vector to store the hidden state. However, the RNN of this structure is prone to have gradient vanishing problem, which is difficult to model for a long time. This issue can be addressed by using LSTM $[1](#References)] and GRU (Gated Recurrent Unit) \[[2](#References)]. This solves the problem of long-term dependency by carefully forgetting the previous information. In this example, we demonstrate the GRU based model. 图 2. GRU 单元 We can see that, in addition to the implicit state, the GRU also contains two gates: the Update Gate and the Reset Gate. At each time step, the update of the threshold and the hidden state is determined by the formula on the right side of Figure 2. These two thresholds determine how the state is updated. ### Bi-directional Encoder In the above basic model, when the encoder sequentially processes the input sentence sequence, the state of the current time contains only the history input information without the sequence information of the future time. For sequence modeling, the context of the future also contains important information. With the bi-directional encoder (Figure 3), we can get both information at the same time: Figure 3. Bi-directional encoder structure diagram The bi-directional encoder \[[3](#References)$ shown in Figure 3 consists of two independent RNNs that encode the input sequence from the forward and backward, respectively. Then it combines the outputs of the two RNNs together, as the final encoding output. In PaddlePaddle, bi-directional encoders can easily call using APIs: wzxhzdk:2 ### Beam Search Algorithm After the training is completed, the model will input and decode the corresponding target language translation result according to the source language. Decoding, a direct way is to take each step conditional probability of the largest word, as the current moment of output. But the local optimal does not necessarily guarantee the global optimal. If the search for the full space is large, the cost is too large. In order to solve this problem, beam search algorithm is commonly used. Beam search is a heuristic graph search algorithm that controls the search width with a parameter $k$] as follows: **1**. During decoding, always maintain $k$ decoded sub-sequences; **2**. At the middle of time $t$, for each sequence in the $k$ sub-sequence, calculate the probability of the next word and take the maximum of $k$ words with the largest probability, combining $k ^ 2$ New child sequence; **3**. Take the maximum probability of $k$ in these combination sequences to update the original subsequence; **4**. iterate through it until you get $k$ complete sentences as candidates for translation results. For more information on beam search, refer to the [beam search](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md#beam-search-algorithm) section in PaddleBook [machine translation](https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation) chapter. ### Decoder without Attention mechanism - In the relevant section of PaddleBook (https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.cn.md), the Attention Mechanism has been introduced. This example demonstrates Encoder-Decoder structure without attention mechanism. With regard to the attention mechanism, please refer to PaddleBook and references \[[3](#References)]. In PaddlePaddle, commonly used RNN units can be conveniently called using APIs. For example, recurrent_layer_group can be used to implement custom actions at each point in the RNN. First, customize the single-step logic function, and then use the function recurrent_group () to cycle through the single-step logic function to process the entire sequence. In this case, the unattended mechanism of the decoder uses recurrent_layer_group to implement the function gru_decoder_without_attention () . Corresponding code is as follows: wzxhzdk:3 In the model training and testing phase, the behavior of the decoder is different: - **Training phase**: The word vector of the target translation trg_embedding is passed as a parameter to the single step logic gru_decoder_without_attention () . The function recurrent_group () loop calls the single step logic execution, and finally calculates the target translation with the actual decoding; - **Testing phase**: The decoder predicts the next word based on the last generated word, GeneratedInput (). The automatic fetch model predicts the highest probability of the $k$ word vector passed to the single step logic. Then the beam_search () function calls the function gru_decoder_without_attention () to complete the beam search and returns as a result. The training and generated returns are implemented in the following if-else conditional branches: wzxhzdk:4 ## Data Preparation The data used in this example is from [WMT14] (http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/), which is a parallel corpus of French-to-English translation. Use [bitexts] (http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/bitexts.tgz) as training data, [dev + test data] (http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/dev+test.tgz) as validation and test data. PaddlePaddle has been packaged in the data set of the read interface, in the first run, the program will automatically complete the download. Users do not need to manually complete the relevant data preparation! ## Model Training and Testing ### Model Training Starting the model training is very simple, just in the command line window to execute python train.py. The train () function in the train.py script of the model training phase completes the following logic: **a) Define the network, parse the network structure, initialize the model parameters.** wzxhzdk:5 **b) Set the training process optimization strategy. Define the training data to read reader** wzxhzdk:6 **c) Define the event handle, print the training intermediate results, save the model snapshot** wzxhzdk:7 **d) Start training** wzxhzdk:8 The output sample is wzxhzdk:9 ### Generate Translation Results In PaddlePaddle, it is also easy to use translated models to generate translated text. 1. First of all, please modify the generate.py script main passed to the generate function parameters to choose which saved model to use. The default parameters are as follows: wzxhzdk:10 2. In the terminal phase, execute the python generate.py command. The generate () in the script executes the following code: **a) Load the test sample** wzxhzdk:11 **b) Initialize the model, execute infer () for each input sample to generate beam search translation results** wzxhzdk:12 **c) Next, load the source and target language dictionaries, convert the sentences represented by the id sequence into the original language and output the results.** wzxhzdk:13 Set the width of the beam search to 3, enter a French sentence. Then it automatically generate the corresponding test data for the translation results, the output format is as follows: wzxhzdk:14 - The first line of input for the source language. - Second ~ beam_size + 1 line is the result of the beam_size translation generated by the column search - the output of the same row is separated into two columns by "\ t", the first column is the log probability of the sentence, and the second column is the text of the translation result. - the symbol  represents the beginning of the sentence, the symbol  indicates the end of a sentence, and if there is a word that is not included in the dictionary, it is replaced with the symbol . So far, we have implemented a basic machine translation model using PaddlePaddle. We can see, PaddlePaddle provides a flexible and rich API. This enables users to easily choose and use a various complex network configuration. NMT itself is also a rapidly developing field, and many new ideas continue to emerge. This example is a basic implementation of NMT. Users can also implement more complex NMT models using PaddlePaddle. ## References [1] Sutskever I, Vinyals O, Le Q V. [Sequence to Sequence Learning with Neural Networks] (https://arxiv.org/abs/1409.3215) [J]. 2014, 4: 3104-3112. [2] Cho K, Van Merriënboer B, Gulcehre C, et al. [Learning phrase representations using RNN encoder-decoder for statistical machine translation](http://www.aclweb.org/anthology/D/D14/D14-1179.pdf) [C]. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014: 1724-1734. [3] Bahdanau D, Cho K, Bengio Y. [Neural machine translation by exclusive learning to align and translate] (https://arxiv.org/abs/1409.0473) [C]. Proceedings of ICLR 2015, 2015