# Taming the Beast: Evaluate, Diagnose, Iterate
Anyone can run a training script. The difference between a trained dragon and a lucky one is **evaluation** — measuring, diagnosing, and iterating like an engineer. This is the lesson that turns your mediocre v1 into a v3 you actually trust, and it's the discipline most hobbyists skip.
## The Three Levels of Evaluation
**Level 1 — Loss curves (free, automatic).** You already have these from training. They tell you about the *process*: did the model learn, did it overfit? They tell you nothing about whether the answers are actually good.
**Level 2 — Your test set (the real exam).** Remember `test.jsonl` — the slice the model has never seen? Run every test prompt through the trained model and grade the outputs against your goal card from lesson 2. This is your ground truth.
**Level 3 — Model-as-judge (scaling up level 2).** Have a strong model (Claude, or your biggest local model) grade each answer against a rubric. Perfect for re-grading 30+ answers on every iteration without losing an evening.
Concept
Grade against your goal card, not against vibes. Ember's card says: structure (answer → explanation → next step), voice, zero invented facts. So the rubric is three yes/no questions per answer. Score = counts, not feelings. "It seems better" is not a measurement.
## Reading the Curves: Three Classic Shapes
**Healthy:** both curves descend, validation slightly above train, both flattening. Ship it to the test set.
**Overfitting:** validation loss bottoms out and climbs while train keeps dropping. The dragon is memorizing. Fixes, in order of power: **more/better data**, fewer epochs (stop where val loss bottomed), lower rank, higher LoRA dropout.
**Undertrained:** both still falling when training ended. Easiest fix in the book: train longer, or raise the learning rate a notch.
## The Diagnosis Table
Numbers point at the problem; reading actual outputs identifies it. After each run, read 10 test outputs and match symptoms:
| Symptom in outputs | Likely cause | Fix |
|---|---|---|
| Ignores your format sometimes | Inconsistent examples, or too few | Clean data; add format-heavy examples |
| Parrots training answers verbatim | Overfitting | Fewer epochs; more data variety |
| Great format, weak substance | Rank too low, or base too small | r=16→32; consider 8B base |
| Invents facts confidently | No refusal examples in data | Add 20–30 explicit refusal demonstrations |
| Worse than base at general chat | Overtrained / catastrophic forgetting | Fewer steps; mix in 10–20% general chat examples |
| Randomly switches language | Mixed-language data without pattern | Make language pairing explicit per example |
Honest Note
Catastrophic forgetting is real: hammer a model with 2,000 support tickets and it may get worse at everything else. LoRA's frozen base limits the damage (delete the adapters and the base is untouched), but adapters can still overpower behavior. The standard vaccine: keep a small slice of general conversation data in the mix.
## The Iteration Loop
Your workflow from here is a loop, and each pass takes an evening, not a week:
1. **Evaluate** v_N on the test set → scores on your goal-card rubric
2. **Read** the 10 worst outputs → pick the *one* biggest problem
3. **Fix the data first** (it's the cause ~80% of the time), hyperparameters second
4. **Retrain** → v_N+1, **re-evaluate same test set** → compare numbers
5. Stop when you hit your goal card's success test — Ember's was ≥25/30 structured, 0 invented facts
Pro Tip
Change one thing per iteration. If you add data, lower the learning rate, and bump the rank all at once, the score moves and you learn nothing about why. Slow is smooth, smooth is fast — this loop converges in 3–4 disciplined passes.
Try It
Run your full test set through v1 and grade it with your rubric — the honest baseline number. Then do exactly one iteration of the loop (probably: add refusal examples and re-clean formatting) and post v2's score next to v1's. That one comparison teaches you more about fine-tuning than any blog post ever will.
Checkpoint
You can read the three loss-curve shapes, diagnose the six classic output symptoms, and you're running a one-change-per-pass iteration loop with real scores. Your dragon is tamed. Next: advanced training — preference tuning, bigger models, and shrinking Ember for deployment.