Machine Translation using Sequence-to-Sequence Learning

Luckily there are quite a few datasets for language translation tasks available here. They all consist of sentence pairs delimited by tabs. The German-English dataset contains  nil↩ sentence pairs prepared by the Tatoeba Project. Before we can feed English sentences to the model, they must first be conformed for use as input to Keras' LSTM layers.

The dataset is already uploaded as deu-eng.txt. The sentence pairs can be loaded by creating a reference (via @...) in the Python runtime.

Let's create a list of lines by splitting the text file at every occurance of '\n'.

Let's look at an example.

Sweet! So we have both the input (English) and the target (German) sentences in every line, separated by '\t'.

Let's go ahead and split each line into input text and target text. Since we'll do the translation character by character we also want to compute a set of every character we encounter in the dataset, both for inputs as well as targets.

Now we can loop over every sample we choose for training and fill the lists and sets. We'll also add '\t' (start-of-sequence) and '\n' (end-of-sequence) characters to every target text. This will later help our model determine when to start and - more importantly - end sequences. We need this due to the fact that we don't know a-priori how long the output sequences should be. That's why we teach our model to decide on that by itself.

Ok, let's look at some characteristics the input and target sequences.

By now we have  nil↩ samples consisting of input/target texts. Along the input texts we have 70 unique characters, while the target texts contain 87 unique characters.

This is, in part, due to these weird German umlauts.

Another characteristic for the German language is that sentences tend to be a bit longer than their English counterparts (maybe you can find a way to test that hypothesis for yourself). In any case, the longest target sequence in the 10000 sample sentences we're using contains 53 characters, while the longest input sequence only contains 16.

But these input texts still don't work as input to our model. We'll need to convert the characters into numeric values. In our case, one-hot encodings are fine, but when turning to more involved models, using more advanced embedding methods such as Word2Vec would make more sense.

Here we first tokenize our characters by assigning each unique character to an integer value.

With that we can start creating numeric data. We'll need input data for both the encoder and the decoder of the model, as well as the target data (used only in the decoder part).

The encoder_input_data will consist of  nil↩ samples of the maximum sequence length (16) filled with the respective one-hot-encoded tokens (in this case a vector of length 70).

The decoder_input_data and the decoder_target_data are both constructed in the same way as the input data for the encoder. We need to construct those two sequences because we're training our model through a process called teacher forcing, where the decoder learns to generate decoder_target_data[t+1...] given decoder_input_data[...t] while taking into account the input sequence via the encoder's internal state. Therefore we have to offset decoder_target_data by one timestep.

Time to fill in the data with the actual tokens. For that we iterate over all input and target texts and insert the respective one-hot encoding each character in the sequence.

With that our example sentence  nil↩ turned into a sequence of length 16 with one-hot encodings of tokens for every character.

Now it's time to take a closer look at our encoder-decoder model. Our model will consist of two LSTMs. One will serve as an encoder, encoding the input sequence and producing internal state vectors which serve as conditioning for the decoder. The decoder, another LSTM, is responsible for predicting the individual characters of the target sequence. Its initial state is set to the state vectors from the encoder. This passes information about what to generate from the encoder to the decoder.

Let's build this model using Keras. For that we'll need the LSTM layer, as well as a Dense layer.

Before building the encoder and decoder parts we first have to define some hyperparameters. Since the dimensionality for the encoder and decoder LSTM layers have to match, one parameter, latent_dim, is fine here.

With that we can start building the model. First we have to construct the encoder. When creating the LSTM layer we have to pay attention to set the return_state argument to true, since we want to use the encoder's internal state vectors for the decoder.

Note that we discard encoder_outputs as depicted in the schematic overview above since we're only interested in the state vectors.

Using encoder_states we can now build the decoder. Again we'll use Keras' Input layer to be flexible concerning input sequence lengths. When creating the LSTM, we now want it to return full output sequences as well as the internal state vectors. We're not using the decoder's internal states during training, but we will need them later for inference.

To arrive at the individual characters from the decoder's output we attach a Dense layer to the decoder's LSTM outputs where the number of units match the number of decoder tokens. This way we can just use a softmax activation for the dense layer's outputs and train the whole model using a categorical cross-entropy loss - a standard choice for classification problems.

Now let's glue both parts together using Keras' Model functional API. Specify the inputs needed to produce the outputs and the resulting model will automatically include all layers necessary to compute the outputs.

Let's compile the model defining the optimizer and our cross-entropy loss. As a cross-check we can also print out a summary of the individual layers included in our model.

No surprises here. lstm_1, our encoder, takes as input the first input layer, while the decoder, lstm_2, uses the encoder's internal states as well as the second input layer. Our model has about 700.000 parameters in total!

Time to run the training! Since we're running a CPU-only instance, the training will take about 1 hour (compared to only about 20 minutes when using a GPU). The patient reader can un-comment the following cell to run the training. We'll skip it and simply load the weights of a pre-trained model.

Since we're impatient by nature, we'll simply use the pre-trained weights. Lets load them.

Ok, with the weights in place it's time to test our machine translation model by translating a few test sentences.

The inference mode works a bit differently than the training procedure. The procedure can be broken down into 4 steps:

1. Encode the input sequence, return its internal states.

2. Run the decoder using just the start-of-sequence character as input and the encoder internal states as the decoder's initial states.

3. Append the character predicted (after lookup of the token) by the decoder to the decoded sequence.

4. Repeat the process with the previously predicted character token as input and updates internal states.

Let's go ahead and implement this. Since we only need the encoder for encoding the input sequence we'll split encoder and decoder into two separate models.

In order to conveniently perform the lookup from step 3 above we'll create reverse-lookup dictionaries for both the input and target tokens.

With that we can create a function to perform the whole process of decoding a given input sequence (inputs already tokenized).

With that let's sample a few test cases!

Not bad for a model consisting only of two LSTM layers and a linear one!

But those were all examples from the training set. Let's validate the model using our own example: Let's have the model translate something simple, like:

"How are you?"

We'll put all the tokenization and decoding into one cell and print out the decoded sequence.

Great! But surely this simple model trained on 10k examples will have some failure cases! Let's try out some examples the model didn't see during training.

Ok, that clearly shows some failure cases. Note that for the evaluation texts we had input sequences longer than the longest sequence in the training set.

While the model is able to produce a decoded sequence for these inputs, it will produce worse outputs the longer the input sequences become. After all, the model was not trained to decode sequences that long. You can try this out by changing the line_ix parameter to something later in the dataset (maybe something around 50000).

This article showed the capabilities of encoder-decoder models combined with LSTM layers for sequence-to-sequence learning.

The dataset consisted of English-German sentence pairs. All characters of the respective sentences were first tokenized. The sequences of tokenized characters were then used as input and target for the encoder-decoder model.

During training we used the teacher forcing method where we offset the decoders input and target by one timestep. For inference, we used a slightly different setup, but still used the same modules we trained earlier. The input sequence was processed by the encoder and its final hidden states, along with the start-of-sequence character, were used as input for the decoder. Each predicted character was then fed back into the decoder while the hidden states were updated. We repeated this until the decoder predicted the end-of-sequence character telling us the predicted sequence is complete.

While this model is a rather simple and far away from state-of-the-art neural machine translation models it still shows some principles still used in more recent approaches (e.g. encoder-decoder models in the infamous Attention Is All You Need paper). Even Google Translate seems to be using a more sophisticated encoder-decoder model based on (bidirectional) LSTMs.

Going further, turning the LSTMs we used into their bidirectional version might help improve the model. To become even better, one might want to turn to methods such as self-attention (described for example in a great blogpost on distill.pub) and embed them in the encoder-decoder structure.