Bias & fairness

What it is

A fruit-ripeness checker learns one rule from the fruit it’s shown: how dark a colour means “ripe.” But apples and bananas ripen at different colours, and the training pile is almost all apples. The checker learns a single line tuned to apples — so it judges apples beautifully and bananas badly. That uneven treatment, caused by a group being under-represented in the data, is what we mean by bias; checking it is fairness.

Go deeper: the model can pick only one “ripeness line.” It lands at an average of the two fruits’ true rules, pulled toward whichever it saw more of. With mostly apples, the line sits near the apple rule and is far too early for bananas, so many half-green bananas get called “ripe.” The striking part: the overall accuracy barely changes, because the few bananas hardly move a score dominated by apples. One number looks fine while a whole group is being failed.

Why care

Real systems have this exact shape. A voice assistant trained mostly on one accent understands it well and others poorly. A skin-condition checker trained mostly on one skin tone is less reliable on the rest. A single “95% accurate” headline can hide a group for whom it’s far worse. Fairness means asking a sharper question than overall accuracy: is the model about as good for every group it will be used on?

The idea, intuitively

Lay each fruit out by colour, with one line marking where the checker says “ripe.” Apples have their own true ripeness mark; bananas have one further along. The checker gets a single line for both. When the training pile is mostly apples, the line snaps to the apples and slices through the bananas in the wrong place — you can see the wrong calls light up. Add bananas to the pile and the line slides to a fair middle.

Peek at the data first

Each fruit has a fruit kind, a colour number, and the true ripe answer. The kind is the group we check fairness across — much like Spectra’s describe_data shows you the make-up of a dataset, and balance can even out a lopsided one.

Try it

Start with the lopsided pile: lots of apples, almost no bananas. Watch the apple bar sit high while the banana bar lags and the red unfairness gap stays wide — even though overall accuracy looks fine. Now slide bananas in the training pile up and watch the one learned line drift right, the banana bar climb to meet the apples, and the gap close.

Where it shows up

• Speech & language. Models trained on a narrow set of voices or dialects work worse for everyone left out of the data.
• Medical & vision tools. A detector trained mostly on one group can be far less accurate on the groups it rarely saw.
• Hiring & lending screens. If past data under-represents or mistreats a group, a model can quietly carry that forward — which is why teams measure accuracy per group.

Where it came from

As machine learning spread into decisions about people in the 2010s, researchers showed that models could be very accurate overall yet much worse for some groups. Studies like Joy Buolamwini and Timnit Gebru’s Gender Shades (2018) found commercial face-analysis systems failed far more often on darker-skinned women — often traced to training data that under-represented them. Work like this turned fairness and per-group evaluation into a standard part of checking a model’s work.

Try it in code

In the Studio, balance evens out a lopsided dataset before training, and check grades the result — the first tools for catching the bias you just saw:

data  = load "fruits"
even  = balance data, by: "type"

train, test = split even, hold_out: 20%
model = make_model "tree"
train_model model, on: train, predict: "type", using: ["sweetness", "size"]

check model, with: test

Open it in the Studio ▶

Check your understanding

• How can a model have high overall accuracy and still be unfair to a group?
• Why does adding more bananas to the training pile close the gap?
• Besides balancing the data, what else might help the checker judge both fruits well?