Anyone can run a training script. The difference between a trained dragon and a lucky one is evaluation — measuring, diagnosing, and iterating like an engineer. This is the lesson that turns your mediocre v1 into a v3 you actually trust, and it's the discipline most hobbyists skip.
The Three Levels of Evaluation
Level 1 — Loss curves (free, automatic). You already have these from training. They tell you about the process: did the model learn, did it overfit? They tell you nothing about whether the answers are actually good.
Level 2 — Your test set (the real exam). Remember test.jsonl — the slice the model has never seen? Run every test prompt through the trained model and grade the outputs against your goal card from lesson 2. This is your ground truth.
Level 3 — Model-as-judge (scaling up level 2). Have a strong model (Claude, or your biggest local model) grade each answer against a rubric. Perfect for re-grading 30+ answers on every iteration without losing an evening.
Concept
Grade against your goal card, not against vibes. Ember's card says: structure (answer → explanation → next step), voice, zero invented facts. So the rubric is three yes/no questions per answer. Score = counts, not feelings. "It seems better" is not a measurement.Reading the Curves: Three Classic Shapes
Healthy: both curves descend, validation slightly above train, both flattening. Ship it to the test set.
Overfitting: validation loss bottoms out and climbs while train keeps dropping. The dragon is memorizing. Fixes, in order of power: more/better data, fewer epochs (stop where val loss bottomed), lower rank, higher LoRA dropout.
Undertrained: both still falling when training ended. Easiest fix in the book: train longer, or raise the learning rate a notch.
The Diagnosis Table
Numbers point at the problem; reading actual outputs identifies it. After each run, read 10 test outputs and match symptoms:
| Symptom in outputs | Likely cause | Fix |
|---|---|---|
| Ignores your format sometimes | Inconsistent examples, or too few | Clean data; add format-heavy examples |
| Parrots training answers verbatim | Overfitting | Fewer epochs; more data variety |
| Great format, weak substance | Rank too low, or base too small | r=16→32; consider 8B base |
| Invents facts confidently | No refusal examples in data | Add 20–30 explicit refusal demonstrations |
| Worse than base at general chat | Overtrained / catastrophic forgetting | Fewer steps; mix in 10–20% general chat examples |
| Randomly switches language | Mixed-language data without pattern | Make language pairing explicit per example |
Honest Note
Catastrophic forgetting is real: hammer a model with 2,000 support tickets and it may get worse at everything else. LoRA's frozen base limits the damage (delete the adapters and the base is untouched), but adapters can still overpower behavior. The standard vaccine: keep a small slice of general conversation data in the mix.The Iteration Loop
Your workflow from here is a loop, and each pass takes an evening, not a week:
- Evaluate v_N on the test set → scores on your goal-card rubric
- Read the 10 worst outputs → pick the one biggest problem
- Fix the data first (it's the cause ~80% of the time), hyperparameters second
- Retrain → v_N+1, re-evaluate same test set → compare numbers
- Stop when you hit your goal card's success test — Ember's was ≥25/30 structured, 0 invented facts