Sentence Comprehension with an LSTM Language Model

The comprehension process for a k-word sentence can be assumed to comprise a sequence of comprehension events for k words: w₁, w₂, ⋯ , w_k or $$ w_1^k$$, or w_1...k for short. After the first t words of the sentence (i.e., $$ w_1^k$$) have been processed, the identity of the upcoming word, w_t+1, is still unknown and can be viewed as a random variable.

The first important concept from the word-information theory is surprisal [8]. It is a measure of the uncertainty about the outcome of a random variable, which is quantified by the probability of the actual next word w_t+1, given the sentence:

Informally, the surprisal of a word can be said to measure the extent to which its occurrence is unexpected.

The language model’s output after processing the sequence $$ w_1^k$$ forms an estimate of $$ P\left(w_{t+1}\left|w_1^t\right.\right)$$ for each possible next word w_t+1, which translates directly into the surprisal of the actual upcoming word.

The second important concept from the word-information theory is entropy [8]. After processing the sequence $$ w_1^t$$, the uncertainty about the remainder of the sentence is quantified by the entropy of the distribution of probabilities over the possible continuations w_{t+1 ... k} (with k>t). The entropy will be formally defined as

where  W_{t+1 ... k} is a random variable with the particular sentence continuations w_{t+1 ... k} as its possible outcomes.

When the next word is encountered, this will usually decrease the uncertainty about the rest of the sentence, that is, H(W_{t+2 ... k}) is generally smaller than H(W_{t+1 ... k}). The difference between the two is the entropy reduction, which will be formally defined

Informally, entropy reduction can be said to quantify how much ambiguity is resolved by the current word, to the extent that disambiguation reduces the number of possible sentence continuations.

Entropy is computed over probabilities of the sentences themselves. That is, the Eq. (2) contains only word sequences instead of structures. As a consequence, there is no more uncertainty about what has occurred up to the current word w_t. Although the intended structure of $$ w_1^k$$ may be uncertain, the word sequence itself is not. This means that only the upcoming input sequence (i.e., from w_t+1 onward) is relevant for entropy.

However, the number of upcoming input sequences is far too large for the exact computation of entropy, even if infinite-length sequences are not allowed and some upper bound on sequences length is assumed. Therefore, probabilities are not estimated over complete sentence continuations. Instead, the look-ahead distance is restricted to some small value n, that is, only the upcoming n words are considered.

In this paper, we consider that entropy is computed over the distribution $$ P\left(w_{t+1}^{t+n}\left|w_1^t\right.\right)$$, which is computed from the LSTM language model’s output by applying the chain rule: 

The definition of entropy from Eq. (2) now becomes

The number of elements in the set W_{t+1 ... t+n} grows exponentially as the n increases. Consequently, the computation time grows exponentially. In this paper, the current simulations do for n = 3 and 4. 

An alternative expression for entropy reduction from Eq. (3) can be simplified as

With preprocessed sentences, the words extracted were encoded into dense word embedding.

In NLP, word embedding is a method of mapping that allows words with similar meanings to have similar representations. We used the Word2vec algorithm, which compresses the dimensions of a word vector from the vocabulary size to the embedding dimension [11-12]. To implement the Word2vec algorithm, we trained it with a total 231005 sentences, with 2,760,125 raw words and 8000 unique words, and with a learning rate of 0.025 decreasing by 0.02, decreasing over 10 epochs.

RNN is a neural network that attempts to model time or sequence dependent behaviour like language.

RNN is performed by feeding back the output of a neural network layer at time  to the input of the same network layer at time t+1. The problem with vanilla RNN is that as we try to model dependencies between words or sequence values that are separated by a significant number of other words, we experience the vanishing gradient problem. This is because small gradients or weights (values less than 1) are multiplied many times over through the multiple time steps, and the gradients shrink asymptotically to zero. This means that the weights of those earlier layers won’t be changed significantly, and therefore the network won’t learn long-term dependencies.

LSTMs are a way of solving this problem for word prediction. An LSTM network has LSTM cell blocks in place of our standard neural network layers. These cells have various components called the input gate (which acts to “kill off” any elements of the input vector that are not required), the forget gate (which helps the network learn which state variables should be “remembered” or “forgotten”), and the output gate (which determines which values are actually allowed as an output from the state cell).

In LSTMs, all the weights and bias values are matrices and vectors, respectively. Input data are represented as Word2vec embedding vector to input words to a neural network, which involves taking a word and finding a vector representation of that word which captures some meaning of the word.

In the section 3.1 to follow, we set up what is called an embedding layer, to convert each word into a meaningful word vector. We have to specify the size of the embedding layer (which the length of the vector each word is represented by). This is usually in the region of between 100-500. In other words, if the embedding layer size is 400, each word will be represented by a 400-length vector( i.e., [x₁, x₂, ⋯, x₃₉₉, x₄₀₀])

In order to get the sentence into the right shape for input into the LSTM language model, each unique word in the text file must be assigned a unique integer index.

As depicted in Fig. 3, the unstructured sentence is first pre-processed in the following three steps: The first step is to transform the letters to lower case, remove punctuations, hyphen, coma, and numbers from the text, and strip any excess white spaces from them. The second step is to tokenize sentences into words. The last step is to save training words and test words into two separate files for usage in the training and test steps.

Fig. 3.

Sentences preprocessing

Basically, the given text file (including sentences with <eos>) was preprocessed into separate words and <unk>. In other words, only the most frequent words (8000 words) were stored in its word dictionary, the others were replaced by the token <unk>, and the words were separated with spaces according to consistent tokenization rules, and the sentences were segmented one per line.

Finally, the original text file was converted into a list of these unique integers, where each word is substituted with its new integer identifier. This allows sentences to be consumed in the neural network.

As in Fig. 2, we convert our words (referenced by integers in the data) into meaningful embedding vectors using the vocabulary dictionary and pre-trained Word2vec model.

The input sequences in the input layer need to be long enough to allow the model to learn the context for the words to predict. To pad input sequences to the same length, we tested the ability of the model to learn with differently sized input sentence. So we built a matrix of (batch size, input size, vocabulary size) where each row corresponds to a word embedding vector and the length of the input sentence is 40 words.

Input sequences that are shorter than 40 words are padded with value (i.e., <pad>) at the end. The longer input sequences are truncated so that they fit the desired length. This layer replaces each word index with a word vector of size (200, 40, 8000).

The model we trained learns to predict the probability distribution for the next word using the context of the preceding words. It predicts the next word with a probability for each word in the vocabulary.

We here specify the type of optimization to perform the given model. The LSTM model is compiled specifying the categorical cross entropy loss needed to optimize the model. For this purpose the efficient Adam optimizer was used.

In order to get good results, we have to run the model over many epochs, and the model needs to have a significant level of complexity to optimize learning parameters. Therefore, we run the model up to 10 epochs and get some reasonable initial results.

Finally, the model was optimized on the training data for 5 training epochs with optimal model parameters. The results are as follows in Table 2 and 3: batch size=200, input size=40, hidden size=512, vocabulary size=8000. After 5 epochs, training accuracy was around 82%, while validation accuracy reached approximately 81%.

Table 2.

Model training with parameters

Table 3.

The performance with accuracy

As depicted in Fig.2, the next word from the previous word given test sentences is predicted in the following two steps. The first step is to preprocess sentences. The second step is to predict the upcoming word in a sentence using the optimal LSTM model and compare the predicted word outputs with the actual word sequences from the test set.

The optimal LSTM model was employed to get the three different word-information values such as surprisal and two entropy-reductions from Eq. (1) and Eq. (4) on each word of a test set of English sentences.

To estimate the sentence processing with the word-to-word prediction using the two measures of surprisal and entropy reduction, we formulated a linear mixed-effect model in the lmer package for R language as follows:

where the term (1|Object) means the random effect variable. The term log(RT) means the log function value of reading-time values for words. Model represents a linear mixed-effects model using the lmer function with one dependent variable log(RT) and fixed effect three variables from Eq. (1) and Eq. (4).