Simple RNN
— msrep 0%
Once upon a time, there was a little girl named Lily. She
next-word candidates
- —
- —
- —
- —
- —
Recurrent Text Lab · Autoregressive generation
ECHO
Four neural networks learned to tell stories from 30,000 children's books. Watch them think — live, in your browser.
live sample · LSTM · T 0.8
Once upon a time, there was a little girl named Lily. She
Chapter 1 · Autoregression
One word at a time.
Every model here plays the same game: given what has been written so far, guess the next word. Then do it again. That loop is what “autoregressive” means — and it is the same idea behind today's largest language models.
- 01
Read
Words become numbers. The model looks only at the last 24 — its context window.
- 02
Predict
It scores all 8,000 words in its vocabulary. Most get almost nothing; a few light up.
- 03
Pick & repeat
One word is chosen, appended, the window slides forward — and the loop runs again.
LSTM · greedystep 01 / 30
Onceuponatime,therewasalittlegirlnamedLily.She
context window (last 24 tokens)
What comes after “She”?
- loved81.4%
- had10.4%
- was2.9%
- liked2.3%
- lived1.9%
Chapter 2 · The library
30,000 bedtime stories, read word by word.
TinyStories is a dataset of short stories written in the vocabulary of a 3–4 year old. Small vocabulary, rich plot and dialogue — exactly the setting where small networks can learn to write coherently. Source on Hugging Face · Eldan & Li, 2023
- Stories
- 30,000
- + 5,000 held out
- Words
- 6.21M
- training tokens
- Vocabulary
- 8,000
- 99.87% coverage
- Training pairs
- 6.2M
- 24 tokens → next token
0150300450600 words
- .512,150
- the283,569
- and253,859
- ,244,966
- to178,211
- a162,780
- "137,779
- was132,623
- he111,384
- she108,903
Once upon a time, in a land full of trees, there was a little cherry tree. The cherry tree was very sad because it did not have any friends. All the other trees were big and strong, but the cherry tree was small and weak. The cherry tree was envious of the big trees. One day,…
Highlighted = input window, violet = the target word. Punctuation counts as a token. Slide one step right and repeat — 6.2 million times.
Chapter 3 · The contenders
Four minds, one rulebook.
Same vocabulary, same 24-word window, same embedding and hidden sizes, same optimiser, same training budget, three random seeds each. The only thing that changes is how each network remembers.
Chapter 4 · Inside the brain
Watch an LSTM decide what to remember.
This is the trained LSTM, running live. Each column is a word it reads; each row is one of its memory neurons. Bright means a gate is open. Click any word to see what the network expected to come next at that moment.
Chapter 5 · The arena
Same seed. Same dice. Four minds.
All models run right here in your browser. Every step they share one random number, so any difference you see comes from the brain — not from luck. Hover any word to see what else the model considered.
0.80 · Balanced
safecreativechaos
Chapter 6 · The scoreboard
Measured, not eyeballed.
Every number is the mean of 3 independent training runs with different random seeds, evaluated on 2,500 stories none of the models ever saw. ± shows the spread between runs.
| Model | Params | |||||
|---|---|---|---|---|---|---|
| GRU | 15.97±0.06best | 42.4%±0.0best | 68.3%±0.0best | 255 | 3.31 | 3.77M |
| LSTM | 17.15±0.04 | 41.5%±0.1 | 67.0%±0.0 | 286 | 0.56best | 4.00M |
| Simple RNN | 23.80±0.11 | 36.6%±0.0 | 62.1%±0.1 | 119best | 2.80 | 3.31M |
| Bidirectional RNN | 23.87±0.16 | 37.4%±0.1 | 62.4%±0.1 | 262 | 5.64 | 5.72M |
Chapter 7 · The creativity dial
Turn up the heat, watch sense melt.
Temperature reshapes the model's probabilities before a word is drawn. Below are 625 pre-generated samples — five seeds, five temperatures, five draws, four models — each scored by an independent judge.
Seed
Temperature
Draw
“Once upon a time, there was a little girl named Lily. She” …
loved to play with his toys and read the ceiling of some colours playing in the park. One day, she would pick up the blocks and make a big mess. When he finished on the crack, the boy had an idea, and told her that Timmy was glad that he was looking for him.
loved to play with her toys and she was very happy to go playing with her friends. One day, she would pick up a red necklace. She said, "Hello, the monkey!" Lily said. "We can play with something lights and give it to my couch and you want to try
loved to play with her toys and read the cherries. One day, her mom said yes and said they were going to visit their neighbours and her birthday. She said, "The sun is bad. We should have to other," mom says. She is glad they have fun. She is sorry
loved to play with her toys and read the ceiling. One day, her mom said she would send it to her dad. She said yes and gave Tommy a treat. Lily felt very happy and excited. She had learned that she had told her all about playing in the couch and that her friends had
GPT-2 perplexity · lower = more natural · brighter = better · human text: 13
| greedy | 0.5 | 0.8 | 1 | 1.3 | |
|---|---|---|---|---|---|
| RNN | 9 | 19 | 35 | 105 | 458 |
| BiRNN | 7 | 15 | 35 | 75 | 329 |
| LSTM | 10 | 14 | 26 | 61 | 220 |
| GRU | 8 | 11 | 23 | 56 | 259 |
Click a cell to load that temperature in the sample browser above.
Closer to the ★ means text that is both as varied and as natural as a human's.
Simple RNNBidirectional RNNLSTMGRU· each line runs greedy (left) → T 1.3 (right)
Chapter 8 · The verdict
The GRU remembers best.
#1
GRU
16.0
test perplexity
−32.9% vs Simple RNN
#2
LSTM
17.1
test perplexity
−28.0% vs Simple RNN
#3
Simple RNN
23.8
test perplexity
#4
Bidirectional RNN
23.9
test perplexity
+0.3% vs Simple RNN
01
Pending results
Findings are written once all training runs complete.