What does return_sequences do in a Keras LSTM?

With return_sequences=False, the default, an LSTM returns only its hidden state after the last time step, shaped (batch, units), which suits classifying a whole sequence. With return_sequences=True it returns the hidden state at every step, shaped (batch, timesteps, units). Every LSTM except the last one in a stack needs return_sequences=True, because the next LSTM expects a sequence.

How do I handle variable-length sequences in Keras?

Pad every sequence to the same length with 0, then set mask_zero=True on the Embedding layer. The mask travels with the tensor to the LSTM or GRU, which skips the padded steps, so a padded sequence produces the same final state as the unpadded one. Without the mask, the recurrent layer processes the padding as if it were real input.

What replaced keras.preprocessing.text.Tokenizer in Keras 3?

Keras 3 removed the old Tokenizer class. Its replacement is the TextVectorization layer, which you fit to a corpus with adapt() and then call on raw strings to get padded integer ids. Because it is a layer, it can live inside the model, so the saved model accepts raw text directly.

Should I use LSTM or GRU?

A GRU has three gates to the LSTM's four, so it has about a quarter fewer parameters and trains slightly faster. On many tasks the two reach similar accuracy. Try both on your data; start with GRU when the model is small or training time matters, and LSTM when sequences have long-range dependencies.

How do I know if a time-series model is actually learning anything?

Compare it against a naive baseline, such as predicting that the next value equals the last one. That baseline costs nothing and is often surprisingly strong. If a trained model cannot beat it on held-out data, the model has not learned useful structure, however low its training loss looks.

TensorFlow RNNs and LSTMs | SkillByExample

Real-World Scenario

A streaming service wants to flag negative reviews for its support team and forecast tomorrow’s viewing load from the last month of traffic. Both inputs are sequences — words in order, readings in order — and recurrent layers read them one step at a time, carrying a hidden state forward. This lesson covers the shapes those layers produce, turning text into padded ids, masking, a sentiment classifier on real movie reviews, and a forecaster checked against a naive baseline.

All examples ran with TensorFlow 2.21 and Keras 3.15 on a CPU-only machine. TensorFlow’s startup log lines on stderr (no CUDA driver found, oneDNN enabled) are left out of the outputs below.

LSTM Inputs and Outputs

A recurrent layer takes (batch, timesteps, features). What comes out depends on two flags.

import numpy as np
import keras

x = np.random.rand(4, 10, 3).astype("float32")     # (batch, timesteps, features)

print(keras.layers.LSTM(8)(x).shape)
print(keras.layers.LSTM(8, return_sequences=True)(x).shape)

seq, h, c = keras.layers.LSTM(8, return_sequences=True, return_state=True)(x)
print(seq.shape, h.shape, c.shape)
print(np.allclose(seq[:, -1, :], h))

$ python lstm_shapes.py
(4, 8)
(4, 10, 8)
(4, 10, 8) (4, 8) (4, 8)
True

By default an LSTM returns only the hidden state after the last step: one 8-number vector per sequence. return_sequences=True returns the hidden state at all 10 steps. return_state=True adds the final hidden state h and cell state c as separate outputs; the last row of the sequence output is exactly h. The cell state is the LSTM’s internal memory and never appears in the sequence output.

SimpleRNN, GRU, and LSTM Side by Side

The three recurrent layers differ in how many gated transformations they learn, which shows up directly in their parameter counts.

import keras

for layer in [keras.layers.SimpleRNN(32), keras.layers.GRU(32), keras.layers.LSTM(32)]:
    model = keras.Sequential([keras.Input(shape=(None, 16)), layer])
    print(f"{type(layer).__name__:10s} {model.count_params():>6,}")
print("per gate:", 16 * 32 + 32 * 32 + 32)

$ python cell_params.py
SimpleRNN   1,568
GRU         4,800
LSTM        6,272
per gate: 1568

Each gate has input weights (16 × 32), recurrent weights (32 × 32), and a bias (32): 1,568 parameters. A SimpleRNN has one such transformation, an LSTM four (input, forget, cell, output: 4 × 1,568 = 6,272), and a GRU three (3 × 1,568 = 4,704) plus 96 more — Keras’s default reset_after=True gives the GRU a second bias vector per gate. shape=(None, 16) leaves the number of time steps open, since the weights do not depend on it.

Turning Text into Padded Ids

TextVectorization learns a vocabulary with adapt, then maps strings to integer ids, padded or cut to a fixed length.

import keras

texts = ["the movie was great", "the plot was dull and slow", "great acting"]
vectorize = keras.layers.TextVectorization(max_tokens=10, output_sequence_length=6)
vectorize.adapt(texts)

print([str(w) for w in vectorize.get_vocabulary()])
print(vectorize(texts).numpy())
print(vectorize(["the popcorn was GREAT!"]).numpy())

$ python vectorize.py
['', '[UNK]', 'was', 'the', 'great', 'slow', 'plot', 'movie', 'dull', 'and']
[[3 7 2 4 0 0]
 [3 6 2 8 9 5]
 [4 1 0 0 0 0]]
[[3 1 2 4 0 0]]

Id 0 is reserved for padding and 1 for unknown words; the rest are ordered by frequency, so was, the, and great — each seen twice — come first. The corpus has nine distinct words but max_tokens=10 leaves room for only eight after the two reserved ids, so acting was dropped and maps to 1. Lowercasing and punctuation stripping are on by default, which is why GREAT! still maps to 4. Keras 3 removed the old keras.preprocessing.text.Tokenizer; this layer replaces it.

Masking Padded Steps

Padding makes a batch rectangular, but the zeros are not real words. mask_zero=True on the Embedding tells the recurrent layer to skip them.

import numpy as np
import keras

keras.utils.set_random_seed(0)
short = np.array([[5, 3, 8]])
padded = np.array([[5, 3, 8, 0, 0, 0]])           # 0 is the padding id

for mask_zero in [False, True]:
    embed = keras.layers.Embedding(10, 4, mask_zero=mask_zero)
    lstm = keras.layers.LSTM(2)
    a = lstm(embed(short)).numpy().round(4)
    b = lstm(embed(padded)).numpy().round(4)
    print(f"mask_zero={mask_zero}: {a} vs {b}")

$ python masking.py
mask_zero=False: [[-0.004  -0.0044]] vs [[0.0075 0.0087]]
mask_zero=True: [[ 0.0131 -0.0044]] vs [[ 0.0131 -0.0044]]

Without a mask, the LSTM runs three extra steps over the embedding of id 0, and the final state changes — even its sign. With the mask, the padded steps are skipped and the output equals the unpadded sequence’s exactly. Each loop iteration creates fresh random weights, so compare the two values on a line, not across lines. The values are small because the weights are untrained.

Sentiment Classification on IMDB

The IMDB dataset ships with Keras as 50,000 movie reviews, already converted to word ids and labelled positive (1) or negative (0). Keep the 10,000 most frequent words, pad or cut each review to 200 ids, and train a bidirectional LSTM.

import time
import keras

VOCAB, MAXLEN = 10_000, 200
(x_train, y_train), (x_test, y_test) = keras.datasets.imdb.load_data(num_words=VOCAB)
print(len(x_train), "reviews; first starts with ids", x_train[0][:6])
x_train = keras.utils.pad_sequences(x_train, maxlen=MAXLEN)
x_test = keras.utils.pad_sequences(x_test, maxlen=MAXLEN)

keras.utils.set_random_seed(0)
model = keras.Sequential([
    keras.Input(shape=(MAXLEN,)),
    keras.layers.Embedding(VOCAB, 32, mask_zero=True),
    keras.layers.Bidirectional(keras.layers.LSTM(32)),
    keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

start = time.perf_counter()
model.fit(x_train, y_train, epochs=2, batch_size=128, validation_split=0.2, verbose=2)
print(f"trained in {time.perf_counter() - start:.0f}s")
print(f"test accuracy: {model.evaluate(x_test, y_test, verbose=0)[1]:.4f}")
model.save("imdb_lstm.keras")

$ python imdb_lstm.py
25000 reviews; first starts with ids [1, 14, 22, 16, 43, 530]
Epoch 1/2
157/157 - 20s - 126ms/step - accuracy: 0.7484 - loss: 0.4858 - val_accuracy: 0.8668 - val_loss: 0.3314
Epoch 2/2
157/157 - 16s - 101ms/step - accuracy: 0.9007 - loss: 0.2528 - val_accuracy: 0.8768 - val_loss: 0.3119
trained in 36s
test accuracy: 0.8681

Two epochs — 36 seconds on a CPU — reach 86.8% on the 25,000 held-out test reviews. pad_sequences pads and cuts at the start by default, so a long review keeps its last 200 words and a short one gets leading zeros, which mask_zero=True then hides. Bidirectional runs one LSTM forwards and a second backwards and concatenates their final states, so the classifier sees context from both ends. Training accuracy has already overtaken validation accuracy in epoch 2 (90.1% vs 87.7%); more epochs would mostly widen that gap.

The ids are word-frequency ranks shifted by 3, with 1 marking the start of a review and 2 an out-of-vocabulary word. get_word_index turns them back into text:

import keras

(x_train, y_train), _ = keras.datasets.imdb.load_data(num_words=10_000)
word_index = keras.datasets.imdb.get_word_index()            # word -> rank
id_to_word = {rank + 3: word for word, rank in word_index.items()}
id_to_word.update({0: "<pad>", 1: "<start>", 2: "<unk>"})

print(" ".join(id_to_word[i] for i in x_train[0][:14]))
print("label:", y_train[0])

$ python decode.py
<start> this film was just brilliant casting location scenery story direction everyone's really suited
label: 1

The dataset is already lowercased and stripped of punctuation. Decoding a few reviews is the quickest check that a preprocessing pipeline has not scrambled the ids.

Time-Series Forecasting

A noisy sine wave stands in for daily traffic. timeseries_dataset_from_array cuts it into 30-step windows, each paired with the value that follows. Saved as series.py, this module is imported by the forecasting scripts.

import numpy as np
import keras

rng = np.random.default_rng(0)
t = np.arange(1200)
series = (10 * np.sin(2 * np.pi * t / 50) + rng.normal(0, 1, t.size)).astype("float32")
WINDOW = 30

def windows(data, shuffle):
    # inputs: data[i : i+WINDOW], target: data[i+WINDOW]
    return keras.utils.timeseries_dataset_from_array(
        data[:-1, None], data[WINDOW:], sequence_length=WINDOW,
        batch_size=32, shuffle=shuffle, seed=0)

train_ds = windows(series[:1000], shuffle=True)
val_ds = windows(series[1000:], shuffle=False)

if __name__ == "__main__":
    x, y = next(iter(val_ds))
    print(x.shape, y.shape)
    print(np.allclose(x[0, :, 0], series[1000:1030]), np.isclose(y[0], series[1030]))

$ python series.py
(32, 30, 1) (32,)
True True

The first validation window is steps 1000–1029 and its target is step 1030 — the check confirms the alignment. The split is by time, not random: the model trains on the first 1,000 steps and is validated on the last 200, so it never sees the future it is asked to predict. data[:-1, None] adds the feature axis and drops the final step, which has no next value to predict.

import numpy as np
import keras
from series import train_ds, val_ds

x_val = np.concatenate([x for x, _ in val_ds])
y_val = np.concatenate([y for _, y in val_ds])
print(f"naive (repeat last value) MAE: {np.mean(np.abs(x_val[:, -1, 0] - y_val)):.3f}")

keras.utils.set_random_seed(0)
model = keras.Sequential([
    keras.Input(shape=(30, 1)),
    keras.layers.LSTM(32),
    keras.layers.Dense(1),
])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])
print(f"parameters: {model.count_params():,}")
history = model.fit(train_ds, validation_data=val_ds, epochs=30, verbose=0)
for epoch in [0, 9, 19, 29]:
    print(f"epoch {epoch + 1:2d}: val_mae {history.history['val_mae'][epoch]:.3f}")

$ python forecast.py
naive (repeat last value) MAE: 1.337
parameters: 4,385
epoch  1: val_mae 5.364
epoch 10: val_mae 0.914
epoch 20: val_mae 0.826
epoch 30: val_mae 0.824

Predicting “the same as yesterday” already scores an MAE of 1.337, so that is the number to beat — after one epoch the LSTM is far worse, at 5.364. By epoch 30 it reaches 0.824. The noise added to the series has a standard deviation of 1, and no model can predict noise: the average absolute size of that noise is about 0.80, so the LSTM is close to the best achievable error. The naive forecast pays for the noise twice — in the last value and in the target — plus the wave’s movement over one step.

Practice

1. Predict the parameter count of LSTM(16) on 8 input features, then check it.

import keras

model = keras.Sequential([keras.Input(shape=(None, 8)), keras.layers.LSTM(16)])
print(model.count_params())
print(4 * (8 * 16 + 16 * 16 + 16))

$ python p1_params.py
1600
1600

Four gates, each with input weights 8 × 16, recurrent weights 16 × 16, and a bias of 16. The count depends on features and units only, never on sequence length.

2. Swap the LSTM in forecast.py for a GRU of the same size. How do the parameter count and final error compare?

import numpy as np
import keras
from series import train_ds, val_ds

keras.utils.set_random_seed(0)
model = keras.Sequential([
    keras.Input(shape=(30, 1)),
    keras.layers.GRU(32),
    keras.layers.Dense(1),
])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])
history = model.fit(train_ds, validation_data=val_ds, epochs=30, verbose=0)
print(f"parameters: {model.count_params():,}")
print(f"final val_mae: {history.history['val_mae'][-1]:.3f}")

$ python p2_gru.py
parameters: 3,393
final val_mae: 0.845

The GRU has 3,393 parameters to the LSTM model’s 4,385 and ends at 0.845 against the LSTM’s 0.824 — both well under the naive 1.337 and near the noise floor. On a task this simple the difference is small; on your own data, measure rather than assume.

3. Load imdb_lstm.keras and score two reviews you write yourself.

import numpy as np
import keras

model = keras.models.load_model("imdb_lstm.keras")
word_index = keras.datasets.imdb.get_word_index()

def encode(text):
    ids = [1] + [word_index.get(w, 999_999) + 3 for w in text.lower().split()]
    ids = [i if i < 10_000 else 2 for i in ids]            # out of vocab -> <unk>
    return keras.utils.pad_sequences([ids], maxlen=200)

for review in ["a brilliant film with wonderful acting",
               "boring and predictable i wanted my money back"]:
    print(f"{model.predict(encode(review), verbose=0)[0, 0]:.3f}  {review}")

$ python p3_own_review.py
0.800  a brilliant film with wonderful acting
0.072  boring and predictable i wanted my money back

The encoding must match the training data exactly: the 1 start marker, ranks shifted by 3, and 2 for any word outside the 10,000 kept. The output is the probability that a review is positive. A mismatch here — forgetting the offset of 3, say — silently produces nonsense scores rather than an error.

4. Stack two LSTMs without return_sequences=True on the first. What error do you get, and what fixes it?

import numpy as np
import keras

x = np.random.rand(4, 10, 3).astype("float32")
try:
    keras.Sequential([keras.Input(shape=(10, 3)),
                      keras.layers.LSTM(32), keras.layers.LSTM(16)])
except ValueError as e:
    print("ValueError:", e)

model = keras.Sequential([keras.Input(shape=(10, 3)),
                          keras.layers.LSTM(32, return_sequences=True),
                          keras.layers.LSTM(16)])
print(model(x).shape)

$ python p4_stack.py
ValueError: Input 0 with name 'None' of layer 'lstm_1' is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: (None, 32)
(4, 16)

The first LSTM returns only its last state, (None, 32), but the second needs a sequence with three dimensions. return_sequences=True hands it the full (None, 10, 32) sequence; only the last LSTM in a stack should return a single vector.