Generative models

What it is

A generative model makes new things in the style of what it learned — new sentences, pictures, or sounds. This one is a tiny Markov chain: it reads a small set of sentences, counts which word tends to follow which, and then “babbles” fresh sentences by rolling weighted dice for each next word.

Go deeper: the chain only remembers the current word when choosing the next one (that’s what “first-order” means). The creativity knob is a temperature: low temperature sharpens the dice toward the most common next word; high temperature flattens them so rarer words can win. Real text generators predict the next token the same way — just with billions of learned numbers instead of a handful of counts.

Why care

Generative AI is the technology behind chatbots, image makers, and music tools. Understanding that it works by predicting what comes next — not by “knowing” or copying — helps you see both its power (endless new combinations) and its limits (it can confidently produce nonsense).

The idea, intuitively

Think of finishing a friend’s sentence. If they say “the happy…” you guess “dog” or “cat” because that’s what usually comes next. Do that one word at a time and you can spin a whole new sentence that sounds right even if nobody ever said it. The chain is just a very literal sentence-finisher.

Peek at the data first

You never type your own text here — sentences come from a small fixed corpus (safety by design). The chain stores, for each word, how often each other word follows it.

Try it

Press Babble a new sentence a few times. Slide Creativity low for safe, common sentences or high for surprising ones, then babble again. Tick Show the next-word odds to see the exact dice the chain rolls after any word.

Where it shows up

• Chatbots. They generate replies one token at a time, predicting what comes next.
• Phone keyboards. Next-word suggestions are a tiny generative model running as you type.
• Image & music tools. Same idea, generating pixels or notes instead of words.

Where it came from

Andrey Markov invented these chains in 1906 and famously used them to model the letters in a Russian poem. Claude Shannon (1948) used word-level chains to generate English-like text in his founding work on information theory. The same “predict the next piece” idea, scaled up with neural networks, powers today’s large language models.

Try it in code

This is exactly the Spectra word-babbler — load a safe corpus, train a Markov model, and generate new sayings:

data = load "sayings"
describe_data data

model = make_model "markov"
train_model model, on: data, using: "text"
generate model, count: 4

Open it in the Studio ▶

Check your understanding

• How can the chain make a sentence that was never in the corpus?
• What does turning creativity (temperature) up actually change?
• Why might a generative model produce something that sounds right but is wrong?