Markov chain
Looks up which tokens followed the same context in the training text.
- Memory
- Transition table
- Context
- Last 1–2 tokens
- Training
- Count once
Markov chain vs. tiny transformer.
Train both on the same text. One counts transitions; one learns weights. Then compare what they write.
Training text
Load an example
Markov chain
Waiting to count transitions.
Tiny transformer
Waiting to train.
How they predict the next token
Counts
Tiny transformer
Uses attention and learned weights to score every token.
- Memory
- Learned numbers
- Context
- Last 32 positions
- Training
- 6 passes, 120–600 updates
Generate with both models
Markov chain
Its counted-path output will appear here.
Next-token odds
Generate to reveal this model's next-token probabilities.
Tiny transformer
Its neural output will appear here.
Next-token odds
Generate to reveal this model's next-token probabilities.
Both models can only use tokens from the training text.
What changed?
| Question | Markov chain | Tiny transformer |
|---|---|---|
| What gets learned? | Counts of observed transitions | Embeddings and matrix weights |
| Can it score an unseen path? | No, it backs off to a shorter seen context | Yes, every vocabulary token gets a score |
| Does word similarity exist? | No, tokens are just table keys | Partly, tokens get learned vectors |
| Where does it run? | Locally in this browser | PyTorch on a Modal T4, with a browser CPU fallback |
| Best teaching insight | Prediction can start with simple statistics | Attention learns which context matters |
How each model learns.
Both predict, sample, and append. They learn in different ways.
Markov chain
lowercase the training text, then split it into tokens
for every token:
count what followed the last 1–2 tokens
to generate:
look up the current context
sample from its counted next tokens
back off if that context was never seenTiny transformer
lowercase the training text, then split it into tokens
give each token and position a learned vector
repeat for 6 passes, bounded to 120–600 updates:
split Q, K, V across 4 attention heads
attention ← softmax(QKᵀ + padding mask) × V
hidden ← 2 × (attention + feed-forward block)
predict the next training token
update every weight using the prediction error
save the weights under a short-lived model ID
to generate on the GPU:
predict and sample one vocabulary token
stream that token back to the browser
append it and repeat