The Lab · Learn

How AI actually works

Small pages that run on this device. Read the plain version, move the sliders, watch what changes, then say what you saw. Tap any word you do not know.

Lesson 4

Evals

Before you swap one prompt or tool for another, test both on the same set of real examples and score them the same way.

It is like: Like comparing two recipes by cooking both for the same guests on the same night, not by reading them.
Why it matters at work: One good-looking answer proves little. A small fixed test set of ten real cases beats a gut feeling every time.

The exact idea

Before you switch prompts, score both on the same fixed set of cases, and know how you are grading.

Words on this page

Try this

With 4 cases, Prompt A wins. Slide to 8 and see who wins now.
Turn off lenient grading. “Delivery.” with a capital and a full stop now fails.
Find the smallest number of cases where B pulls ahead.

Change it

Ask for a way to add your own test case, with a message, the right label and what each prompt said.

Prompt A wins on 4 cases and B wins on 8. What does that tell you?

Pick the one that fits what you saw.

B is better, because 8 cases is the bigger test.Neither result means much yet. A few cases are noisy, and you need more cases that look like real use.A is better, because it won on the first test.The grader is broken, because the better prompt should win every time.

Field notes

Reading the notes.