The Lab · Learn

How AI actually works

Small pages that run on this device. Read the plain version, move the sliders, watch what changes, then say what you saw. Tap any word you do not know.

Lesson 11

Hold some data back

To find out how good a model really is, test it on examples it never saw while it was learning.

It is like: Like a driving test on roads the learner has never practiced on. Practice roads only prove they learned those roads.
Why it matters at work: When a vendor quotes accuracy, ask what it was measured on. Numbers on the training examples always look better than real life.

The exact idea

To judge a model honestly, score it on data it was not fitted to, because training error only ever looks better.

Words on this page

Try this

Slide the degree from 0 to 10. The solid training line keeps falling.
Watch the dashed validation line fall, then climb. The dot marks its lowest point.
Press “Hold back different points” a few times and see how much the best degree moves.

Change it

Ask for a slider that sets how much data is held back, from 10% to 50%, and watch the dot move.

Why not pick the degree with the lowest training error?

Pick the one that fits what you saw.

The training error is noisy, so its lowest point is just luck.Training error keeps falling as the model bends closer to the points it is scored on, so it would always pick the most flexible model.Validation error is always lower, so it is the more flattering number.A lower training error takes longer to train, so a middle degree is a better trade.

Field notes

Reading the notes.