Journal Archive

LMs reflect the syntactic and semantic regularities of a given language for recognition or generation. Generally, LMs calculate a probability of a word sequence to measure how likely the sequence(or a sentence) is. Given a sentence (w₁, w₂, ...,w_T), the probability is calculated by factorized conditional probabilities as follows:

where w_t is the t-th word, and w1t-1 stands for a word sequence (w₁, ...,w_t-1)

A neural network with one-hot vectors as its input can learn the word embedding vectors [14]. In neural network based language modeling (NNLM), the conditional probabilities are implemented by forward propagation in neural networks, and the first NNLM was based on feed forward neural network [15], which generalizes to unseen or rare n-grams. To overcome the limitation of the Markov assumption in feed forward neural network LMs, RNNLM has been proposed [13]. The ordinary RNN computes the output sequence of hidden node from an input sequence, which is obtained by

where Φ can be implemented as long short-term memory (LSTM [16]), gated recurrent units (GRU[17]) or any other RNN units [18], and x_t is the word embedding of w_t.

where 1 (w_t)_j is a one-hot vector defined as

E∈RE×V is a word embedding matrix, where E and |V| are the word embedding dimension and the vocabulary size, respectively.

Finally, the output of the neural networks is a probability distribution for the next word

where W_i is the i-th row vector of the matrix, W∈RV×E and b_i is bias.

The LM model is usually trained to maximize the log-probability of the correct prediction of the next word, given the previous words in the sequence using a large training parallel corpus. This is done by stochastic gradient de- scent. It has been observed that these unsupervised word embedding vectors can be used to greatly improve supervised natural language tasks [19, 20]

The attention-based NMT system computes a conditional distribution over translations given a source sentence X=w1x,w2x,⋯,wTx:

This is done by a neural network that consists of an encoder and a decoder. The encoder is often implemented as a bidirectional recurrent neural net- work that reads the source sentence word-by-word. Before being read by the encoder, each source word wtx∈V is projected onto a continuous vector space:

where Ex∈RE×V is a source word embedding matrix. The resulting sequence of the word embedding vectors is then read by the bidirectional encoder recurrent network which consists of forward and reverse recurrent networks.

The decoder consists of two sub-components–a recurrent network and the attention mechanism [21]. The recurrent network in the decoder is unidirectional, which computes the conditional distribution over the next target word, given all the previous target words and the source sentence:

The decoder recurrent network maintains an internal hidden state z_t. At each time step t′, it first uses the attention mechanism to make a context vector, c_t^,, from the annotation vectors that are the output of the encoder. The decoder recurrent network updates its own hidden state by

Like the encoder, Φ_z can be implemented as either an LSTM or GRU. y_t^′-1 is a target-side word embedding vector computed by

similarly to Eq. (7). The probability of each word i in the target vocabulary V′ is computed by

As in LM, the NMT model is usually trained to maximize the log-probability of the correct translation, given a source sentence, which is done by stochastic gradient descent.

We evaluated the proposed word embedding transfer on two translation tasks; (1) En-De and (2) En-Fi. As a baseline task, for each language pair, we used all the parallel corpora available from WMT’16 for training, which results in 4.5M and 2M sentence pairs for En-De and En-Fi, respectively. In the case of En-De, we preprocessed the parallel corpora following [22] and ended up with 100M words on the English side. For En-Fi, we did not use any preprocessing routine other than simple tokenization. Instead of space- separated tokens, we use 30k subwords extracted through byte pair encoding (BPE), as suggested in [11]. When computing the translation quality using BLEU, we un-BPE’d the resulting translations, but left them tokenized.

For word embedding transfer, we trained three LMs for three languages: En, De, and Fi. Note that we have one LM for En, although we transfer the same word embedding matrix to the two NMT models for En-De and En-Fi, respectively. We used monolingual data from the WMT’16 page as in Table 1. Since the monolingual data is not abundant for En-Fi, we used both monolingual data and parallel data for the corresponding LMs. Note that the corpus of Fi for LM is twice as large as the parallel corpus. For En-De, the parallel corpus was not included for LM training, since the monolingual data of De was large enough, compared to Fi.

Once a model is trained, we use a simple forward beam search with the width set to 12 to find a translation that approximately maximizes log p (Y|X). The decoded translation is then un-BPE’d and evaluated against a reference sentence by BLEU (in practice, BLEU is computed over a set of sentences). We use ‘newstest2013’ and ‘newstest2015’ as the validation and test sets for En-De, and ‘newsdev2015’ and ‘newstest2015’ for En-Fi.

We present the translation qualities of all the models on both En-De and En-Fi in Table 2. For En-De, the differences are marginal. However, for En-Fi, whose corpus is much smaller than En-De, the table shows significant improvement in the translation qualities.

Also, the model size to train is 38% compared to the baseline model, which means training per each iteration is faster and converges faster than the baseline model. Note that the data set for NMT of De is completely different from the LM data. Yet, the proposed method has almost the same performance with a much smaller model size.