Ember flies and follows orders. This lesson is the advanced flight school: training bigger dragons on the same hardware (QLoRA), teaching judgment instead of imitation (DPO), and shrinking your trained model for fast deployment (quantization + GGUF). These three techniques are what separate weekend experiments from models you run every day.
QLoRA: Training Above Your Weight Class
You already used 4-bit loading in lesson 5 on the PC path — that was QLoRA. Now let's use it deliberately, for its real purpose: training models that don't otherwise fit.
The idea: compress the frozen base weights to 4-bit precision (NF4 format), keep the LoRA adapters in 16-bit. The base loses a sliver of quality from compression, but since the adapters — the part that's learning — stay full precision, final results land remarkably close to full LoRA.
From lesson 3's table, QLoRA is what unlocks each machine's ceiling:
- RTX 4090 (24 GB): 14B comfortably, 32B just barely (short context, batch 1, gradient checkpointing — expect a fight).
- M3 Ultra (96 GB): 32B with room to spare; 70B is possible if you accept overnight runs.
When should Ember upgrade from 4B? Only when evaluation says so: if v3+ keeps failing on substance (weak reasoning in complex answers) while format and tone are solid, that's the "base model too small" symptom from lesson 6 — jump to Qwen3-8B or 14B and rerun the same pipeline. Nothing else changes; that's the beauty of the setup.
Pro Tip
Scaling from 4B to 8B roughly doubles training time and memory — usually worth it. 14B → 32B doubles it again for a smaller quality jump on simple tasks. Let your test scores justify each jump, not model-size bragging rights.DPO: Teaching Taste, Not Just Tricks
Supervised fine-tuning (everything so far) shows the model good answers. DPO — Direct Preference Optimization — shows it pairs: for the same prompt, a chosen answer and a rejected one. The model learns the direction of your preferences: what makes one answer better than another.
{"prompt": "Customer: You people are useless! Where is my package?!",
"chosen": "I understand the frustration — let's fix this properly. [calm, structured answer with next step]",
"rejected": "We apologize for any inconvenience caused. Your satisfaction is important to us. [corporate mush, no next step]"}
Concept
SFT teaches "here is what a good answer looks like". DPO teaches "between these two plausible answers, prefer this kind". It shines exactly where SFT plateaus: subtle style judgment, de-escalation, choosing brevity, avoiding corporate filler — things that are hard to demonstrate but easy to compare.The recipe order matters: always SFT first, then DPO on top — typically 100–500 preference pairs, with a lower learning rate (~5e-6). The best rejected answers aren't strawmen; they're your own model's real, plausible-but-flawed outputs from evaluation runs. On the PC path this is trl's DPOTrainer with the same Unsloth setup; on the Mac, mlx-lm ships preference-training support (check mlx_lm.lora --help for the current flags — the ecosystem moves fast).
Honest Note
DPO is a seasoning, not a meal. On a small model with a solid SFT, 200 good pairs noticeably sharpen tone judgment. DPO cannot rescue a bad SFT, and overdoing it (too many epochs, too high a learning rate) produces a model that's confidently weird. If you only have one evening, spend it on better SFT data.Quantization: Shrinking the Dragon for Daily Flight
Training precision and serving precision are different games. Once Ember is final, you'll want it small and fast for everyday use. That's quantization at inference time, and the universal format for it is GGUF — the format Ollama, LM Studio, and llama.cpp all speak.
The pipeline, on either machine:
Step one is merging — folding your adapters into the base weights so you get a single standalone model:
# Mac (MLX)
mlx_lm.fuse --model Qwen/Qwen3-4B-Instruct \
--adapter-path adapters/ember-v3 --save-path ember-merged
# PC (Unsloth) — can even save GGUF directly:
model.save_pretrained_merged("ember-merged", tokenizer)
model.save_pretrained_gguf("ember-gguf", tokenizer, quantization_method="q4_k_m")
Q4_K_M is the community's default sweet spot: ~4.5 bits per weight, quality loss barely measurable on most tasks, size cut ~4× vs 16-bit. Use Q8_0 when you want near-lossless and have the memory; go below Q4 only for the memory-desperate.