# Advanced Dragon Training: QLoRA, DPO, and Quantization Ember flies and follows orders. This lesson is the advanced flight school: training *bigger* dragons on the same hardware (QLoRA), teaching *judgment* instead of imitation (DPO), and shrinking your trained model for fast deployment (quantization + GGUF). These three techniques are what separate weekend experiments from models you run every day. ## QLoRA: Training Above Your Weight Class You already used 4-bit loading in lesson 5 on the PC path — that *was* QLoRA. Now let's use it deliberately, for its real purpose: **training models that don't otherwise fit.** The idea: compress the frozen base weights to 4-bit precision (NF4 format), keep the LoRA adapters in 16-bit. The base loses a sliver of quality from compression, but since the adapters — the part that's learning — stay full precision, final results land remarkably close to full LoRA. From lesson 3's table, QLoRA is what unlocks each machine's ceiling: - **RTX 4090 (24 GB):** 14B comfortably, 32B just barely (short context, batch 1, gradient checkpointing — expect a fight). - **M3 Ultra (96 GB):** 32B with room to spare; 70B is possible if you accept overnight runs. When should Ember upgrade from 4B? Only when evaluation says so: if v3+ keeps failing on *substance* (weak reasoning in complex answers) while format and tone are solid, that's the "base model too small" symptom from lesson 6 — jump to Qwen3-8B or 14B and rerun the same pipeline. Nothing else changes; that's the beauty of the setup.

Pro Tip

Scaling from 4B to 8B roughly doubles training time and memory — usually worth it. 14B → 32B doubles it again for a smaller quality jump on simple tasks. Let your test scores justify each jump, not model-size bragging rights.
## DPO: Teaching Taste, Not Just Tricks Supervised fine-tuning (everything so far) shows the model *good* answers. **DPO — Direct Preference Optimization** — shows it *pairs*: for the same prompt, a **chosen** answer and a **rejected** one. The model learns the *direction* of your preferences: what makes one answer better than another. ```json {"prompt": "Customer: You people are useless! Where is my package?!", "chosen": "I understand the frustration — let's fix this properly. [calm, structured answer with next step]", "rejected": "We apologize for any inconvenience caused. Your satisfaction is important to us. [corporate mush, no next step]"} ```

Concept

SFT teaches "here is what a good answer looks like". DPO teaches "between these two plausible answers, prefer this kind". It shines exactly where SFT plateaus: subtle style judgment, de-escalation, choosing brevity, avoiding corporate filler — things that are hard to demonstrate but easy to compare.
The recipe order matters: **always SFT first, then DPO on top** — typically 100–500 preference pairs, with a lower learning rate (~5e-6). The best rejected answers aren't strawmen; they're your *own model's* real, plausible-but-flawed outputs from evaluation runs. On the PC path this is `trl`'s `DPOTrainer` with the same Unsloth setup; on the Mac, `mlx-lm` ships preference-training support (check `mlx_lm.lora --help` for the current flags — the ecosystem moves fast).

Honest Note

DPO is a seasoning, not a meal. On a small model with a solid SFT, 200 good pairs noticeably sharpen tone judgment. DPO cannot rescue a bad SFT, and overdoing it (too many epochs, too high a learning rate) produces a model that's confidently weird. If you only have one evening, spend it on better SFT data.
## Quantization: Shrinking the Dragon for Daily Flight Training precision and serving precision are different games. Once Ember is final, you'll want it **small and fast** for everyday use. That's quantization at inference time, and the universal format for it is **GGUF** — the format Ollama, LM Studio, and llama.cpp all speak. The pipeline, on either machine: Merge adapters base + LoRA → one model Convert to GGUF llama.cpp convert script Quantize Q4_K_M ~8 GB → ~2.5 GB (4B) Serve Ollama · LM Studio Step one is merging — folding your adapters into the base weights so you get a single standalone model: ```bash # Mac (MLX) mlx_lm.fuse --model Qwen/Qwen3-4B-Instruct \ --adapter-path adapters/ember-v3 --save-path ember-merged ``` ```python # PC (Unsloth) — can even save GGUF directly: model.save_pretrained_merged("ember-merged", tokenizer) model.save_pretrained_gguf("ember-gguf", tokenizer, quantization_method="q4_k_m") ``` **Q4_K_M** is the community's default sweet spot: ~4.5 bits per weight, quality loss barely measurable on most tasks, size cut ~4× vs 16-bit. Use Q8_0 when you want near-lossless and have the memory; go below Q4 only for the memory-desperate.

Try It

Take your best Ember, merge, convert, quantize to Q4_K_M — then run your lesson-6 test set on the quantized model. Compare scores with the unquantized version. Seeing (numerically!) that Q4_K_M barely moved your metrics is how you earn the confidence to deploy small.

Checkpoint

You know when QLoRA unlocks bigger bases, when DPO adds judgment on top of SFT, and how merge → GGUF → Q4_K_M turns a training artifact into a deployable model file. One thing left: releasing your dragon into the world.