The Lab · Learn

How AI actually works

Small pages that run on this device. Read the plain version, move the sliders, watch what changes, then say what you saw. Tap any word you do not know.

Lesson 12

Gradient descent

Training is many small adjustments, each one making the model a little less wrong, with a setting for how big each step is.

It is like: Like walking down a foggy hill by feel. Each step goes whichever way slopes down. Steps too big and you overshoot the valley.
Why it matters at work: It explains why training takes time and money, and why a model can settle on an answer that is good but not the best.

The exact idea

Training moves a model's numbers a little way downhill on the loss each step, and the learning rate sets how far.

Words on this page

Try this

At rate 0.10, take 10 steps. The ball creeps down toward the bottom from one side.
Set the rate to 0.80 and step. It zigzags across the bottom but still settles.
Set the rate to 1.10 and take 10. Each step lands further up the other side: it diverges.

Change it

In the Workshop, press Pick a part, then tap the learning rate slider and ask for it to go up to 2, then take 10 steps and read the loss.

Why does a learning rate that is too large make the loss grow instead of shrink?

Pick the one that fits what you saw.

A large rate bends the loss curve into a different shape.The ball builds up speed and cannot stop at the bottom.Each step jumps past the bottom and lands higher on the other side, where the slope is steeper, so the next step is bigger.A large rate turns the steps uphill instead of downhill.

Field notes

Reading the notes.