Fine-tuning & Alignment

Fine-tuning changes a model's weights. Use it when you need to reshape behavior, format, style, or task skill — not when you merely need fresher facts.

The adaptation ladder is prompt → RAG → fine-tune → distill. Climb only when the cheaper rung cannot deliver the behavior you need.

Mental model

Adaptation changes a model's behavior distribution using examples, preferences, or compressed teacher signals. The objective, reference model, parameter surface, dataset, and evaluation contract determine what changes and what may be forgotten; new factual context often belongs in retrieval instead.

Roadmap: decision frame and training methods

• When to fine-tune vs prompt vs RAG
• Data quality > quantity
• Building the fine-tuning dataset
• Evaluating a fine-tune

Training methods

• Supervised fine-tuning / instruction tuning
• LoRA and adapters
• QLoRA and 4-bit fine-tuning
• RLHF with PPO, conceptually
• Direct Preference Optimization

Failure modes and deployment economics

• Catastrophic forgetting
• Distillation
• Cost and hardware

Connects to: Data for AI · Reinforcement Learning · Evaluation

Core sources

• LoRA — low-rank parameter-efficient adaptation.
• QLoRA — quantized base weights with trainable adapters.
• InstructGPT — supervised instruction tuning and RLHF pipeline with human evaluation.
• Direct Preference Optimization — preference optimization without an explicit online RL loop.
• Hugging Face PEFT — current implementation semantics for adapters and parameter-efficient methods.