Direct Preference Optimization (DPO): End-to-End Implementation

In previous video, we derived DPO from scratch, why log probability ratios are the implicit reward, how the Bradley-Terry loss becomes a simple binary cross-entropy, and why you don't need PPO. That was the theory. Today, we build it end-to-end. Dataset construction, cleaning, formatting, because bad preference data is the number one reason DPO runs fail, and nobody talks about it. Then, the full training pipeline with TRL and accelerate, including multi-GPU setup. Then, evaluation. What metrics actually tell you if DPO worked? And finally, a direct code-level comparison with PPO, so you can make an informed choice for your specific project. Let's build. Step one, and the step most tutorials skip, is the dataset. Bad preference data is the number one reason DPO training fails. Before you write a single line of training code, you need to get the dataset right. TRL's DPO trainer expects a dataset with exactly three fields: prompt, chosen, and rejected. You can use raw strings or chat template-formatted message lists. Both work. The chat template format is preferred for instruct models because it applies the model's tokenizer chat template automatically. Now, the quality checks. These are not optional. First, filter near-tie pairs. If human annotators were divided on a pair, one chose A, another chose B, the label is noise. Drop any pair where inter-annotator agreement was below 70%. For AI-labeled data, filter pairs where the judge's confidence score was below 0.6. Second, length balance check. Compute the Pearson correlation between response length and preference label. If this is above 0.3, your annotators or your AI judge are systematically preferring longer responses. This creates a length bias. DPO will learn to generate longer responses, not better ones. Filter pairs where chosen is more than twice the length of rejected. Third, minimum and maximum token filtering. Responses under 20 tokens are usually truncated or incomplete and don't carry useful preference signal. Set a sensible maximum, typically 1024 tokens for standard training. Fourth, deduplication by prompt. If the same prompt appears with multiple preference pairs, keep only the highest confidence pair. Duplicate prompts with conflicting signal confuse the gradient. Here's the data set preparation code. I'm using Anthropic's It's 170,000 helpfulness and harmlessness pairs, freely available. The format pair function extracts the prompt and the final assistant responses from HH's conversation format. HH stores the full conversation in a single string. We split at the last human-assistant boundary to isolate the prompt from the final response. Then three quality filters. Minimum 10 words per response, anything shorter is usually incomplete. Maximum 512 words. Longer responses need a truncation strategy we'll configure in the trainer. And a basic check that chosen doesn't equal rejected. Surprisingly common in large scraped datasets. The length bias diagnostic is critical and most implementations skip it. Compute the Pearson correlation between whether chosen is longer than rejected and the length difference. A correlation below 0.2 means your data set doesn't have a systematic length preference issue. Above 0.4 is a red flag. Your model will learn to generate longer responses regardless of quality. For public data sets, HHRLHF is a solid starting point for general helpfulness. Ultra feedback binarized gives you 200,000 AI labeled pairs across instruction following, math, and code. Help steer two from Nvidia is 21,000 expert labeled pairs with the highest quality signal. If you're training a code model, ultra interact has reasoning and coding pairs specifically. Here's the full training pipeline, five steps. Step one, load the policy model in four-bit NF4 quantization. We're using QLoRA style loading, bits and bytes four-bit with BF16 compute D type. This brings a Llama 3.1 8B model into about 5 GB of VRAM instead of 16 GB. Step two, wrap the policy with LoRA adapters. For DPO, I recommend a higher rank than you'd use for SFT. R = 64 with alpha 128. The reason DPO makes subtle adjustments to the probability distribution. A higher rank adapter gives it more expressive capacity to make these adjustments precisely. Apply it to all four attention projection matrices, Q, K, V, and output. Step three, load the reference model. This is a second copy of the SFT checkpoint. No LoRA, no training. It stays frozen throughout. Its log probabilities are the anchor that defines the implicit reward. Using four-bit loading here keeps it memory efficient. Step four, configure DPO config. The critical settings, beta = 0.1 as our standard default, learning rate 5e to the 7th, extremely low because you're adjusting an already well-trained model, effective batch size of 16 through gradient accumulation, two per device times eight accumulation steps, max length 1024 with max prompt length 512, gradient checkpointing to save activation memory, warm-up ratio 0.1, 10% of steps for warm-up. Step five, instantiate DPO trainer with both models and call train. TRL handles the four log probability forward passes per batch, chosen under policy, rejected under policy, chosen under reference, rejected under reference, computes the log ratios, computes the DPO loss, and back propagates only through the policies LoRA adapters. Let's cover multi-GPU setup because most serious DPO runs need more than one GPU. With accelerate, you go from a single GPU run to multi-GPU with two changes, an accelerate config.yaml and changing your launch command to accelerate.launch. No changes to the training code itself. The YAML specifies distributed type multi-GPU, number of processes, and the mixed precision. For DPO specifically, you'll typically use FSDP, fully sharded data parallel, because you have two large models to hold simultaneously, the policy and the reference. FSDP sharding strategy one is full shard, which distributes both model parameters and optimizer states across all GPUs. This is more memory efficient than DDP for the two model DPO setup. Memory numbers for an 8B model on a single RTX 4090 with QLoRA loading, 4-bit policy plus 4-bit reference plus Lora adapters and optimizer states fits in about 20 GB, just within the 24 GB budget. This is the single GPU sweet spot. On two A100 80 GB with full BF16 Lora DPO, no 4-bit quantization, you use about 38 GB per GPU, well within budget and faster per step than Q-Lora. On four A100s with FSDP, each GPU holds about 22 GB and training is substantially faster thanks to parallelism. One TRL specific note on multi-GPU, when you pass both the policy model and ref model to DPO trainer, TRL automatically handles sharding both models with FSDP. If you're on a memory constrained setup, you can offload the reference model to CPU, pass ref model in init key wargs with device map equals CPU, and TRL will move it to GPU only when needed for the reference log probability forward pass. DPO training emits a rich set of metrics that most people don't use fully. Let me walk through what each one tells you. The four rewards metrics are the most important. Rewards/chosen is the implicit reward for chosen responses. It should increase over training as the policy assigns higher log prog ratio to preferred responses. Rewards/rejected should decrease. Rewards/margins is the gap between them. This is your primary signal that DPO is actually working. Rewards/accuracies is the fraction of batches where choosing margin exceeds rejected margin. Target above 70% for a healthy run. Log PS/chosen and log PS/rejected show the raw log probabilities from the policy, useful for diagnosing specific failure modes. The approximate KL divergence tells you how far the policy has drifted from the reference. This should grow slowly and smoothly. If it spikes, the policy is diverging from the reference faster than the preference signal can guide it. Increase beta. Three patterns to recognize. Healthy. Accuracy rising steadily from 55% towards 70 plus percent. Margin growing, loss declining, both log PS moving in the right direction. Diverging. Accuracy plateaus around 55%. This almost always means data set quality issues or the learning rate is too high. Collapsing. Log PS/rejected drops to very large negative values, while log PS/chosen barely moves. The model has learned to suppress rejected responses rather than improve chosen ones. The fix is simple. Add a standard language modeling NLL loss on the chosen responses alongside the DPO loss. The total loss is DPO loss plus lambda times NLL. The NLL term forces the model to maintain fluency on preferred responses, while the DPO term drives the preference ordering. Lambda, called RPO alpha in TRL, controls the balance. In TRL, you enable this with a single parameter, RPO alpha. Setting it to 1.0 gives equal weight to DPO and NLL. This is the robust preference optimization setting, and it's what most production runs should use. It increases training stability significantly, especially on noisier data sets like AI labeled preference data. Evaluation is where DPO implementations diverge most in quality. Let me give you a three-tier framework. Tier one, standardized offline benchmarks. MT-Bench is the most widely used. A GPT-4 judge scores responses to 80 multi-turn questions across coding, reasoning, writing, and role play on a 1 to 10 scale. Alpaca Eval 2.0 measures win rate against GPT-4 Turbo on 800 instruction-following prompts. Eiffel Eval measures whether the model follows explicit formatting instructions. Run these before and after and target at least two points of MT-Bench improvement and a positive Alpaca Eval win rate delta. Tier two, AI judge win rate against your SFT baseline. This is faster to run than standardized benchmarks and lets you iterate quickly. Generate responses from both your SFT model and your DPO model on a held-out set of prompts from your specific use case domain domain. Then ask GPT-4o or Claude to judge which response is better in a pairwise comparison. A win rate above 55% versus the SFT model indicates meaningful improvement. If you're below 55%, the DPO run may not have moved the needle on your specific task. Tier three, capability regression monitoring. This runs during training, not after. Use LM evaluation harness to run MMLU, HellaSwag, or whatever general capability test your SFT model was strong on. Check it every 500 steps. If it drops more than three points from the SFT baseline, stop training, increase beta, and restart. Let me show you the infrastructure difference side by side in actual code because the abstract comparison doesn't capture how dramatically different the operational burden is. DPO, two models, policy and frozen reference, both loaded from the SFT checkpoint. Static preference data set, standard gradient descent through DPO trainer, no reward model training, no RL loop, no sampling. Training time for an 8B model on 50,000 pairs with 4 A100s, roughly 8 hours. PPO, four models, policy, frozen reference, reward model, and value model. You have to train the reward model first. That's a separate training run taking several more hours. Then the PPO learning loop involves active sampling. On each step, you sample responses from the current policy, score them with the reward model, compute advantages, and run a PPO update step. The sampling, scoring, updating loop is much slower than standard gradient descent. Training time for the same setup, 32 hours or more, plus the reward model training time. The operational overhead compounds when you account for the fact that PPO has significantly more hyperparameters, clip ratio, value function coefficient, GAE lambda, rollout batch size, PPO epochs per rollout. All of these interact with each other and require careful tuning. DPO has effectively one critical hyperparameter, beta. The break-even, PPO's advantage becomes meaningful when you can generate new preference data from the current policy during training, the online learning benefit. If you're using a static preference data set, DPO gives you equivalent or better results at a quarter of the compute cost. Let me give you the actual benchmark numbers because this is what should drive the DPO versus PPO decision. On MT-Bench, measuring instruction following quality, SFT baseline is 6.8. DPO reaches 7.6, a 0.8 point gain. PPO reaches 7.8, a 1.0 point gain. DPO is 80% of PPO's gain. On Alpaca Eval 2.0 win rate versus GPT-4 Turbo, SFT at 18%, DPO at 26%, an 8 percentage point improvement. PPO at 28%, 10 points. Again, DPO captures about 80% of PPO's gain. The capability regression numbers are where DPO's real advantage shows. On MMLU, the general knowledge benchmark, SFT at 63%, DPO drops it by one point to 62%. PPO drops it three points to 60%. On HumanEval for code, DPO drops it by one point, PPO drops it by five points. DPO consistently shows better preservation of SFT capabilities during alignment training. Harmful content refusal, DPO improves from 72% to 88% on a standard safety valve, PPO to 90% comparable. Training compute, DPO adds about half an SFT run in additional compute. PPO adds three to five SFT runs worth of compute including the reward model training. The meta finding, DPO achieves 80 to 85% of PPO's alignment game at 20 to 30% of the compute cost with better preservation of general capabilities. For the vast majority of alignment tasks, this trade-off favors DPO strongly. I want to be clear about when PPO is genuinely worth the infrastructure cost because DPO's strengths don't mean PPO is obsolete. Case one, you have an online feedback loop. If you can score new responses from the current policy during training using a reward model, an AI judge, or a programmatic checker, PPO's online learning advantage compounds over time. DPO is limited to a static data set of responses generated by the SFT model. As the policy improves during DPO training, those static responses become less representative. PPO trains on its own current outputs and avoids this distribution shift entirely. Case two, you have verifiable reward signals. If your task has a programmatic correctness check, code that either passes unit tests or doesn't, math problems with exact correct answers, SQL queries that either return the right rows or don't, PPO with this hard reward signal dramatically outperforms DPO. This is how DeepSeekR 1 and OpenAI's O3 were built. The exact signal from an execution environment is far stronger than the statistical signal from preference pairs. Case three, frontier scale alignment. The models you interact with from major labs, GPT-4, Claude, Gemini, use PPO or PPO variants because they're training on enormous preference data sets with large human evaluation teams. The quality bar requires the additional alignment gains from online learning and the infrastructure cost is amortized over billions of users. The practical escalation path, start with DPO. If DPO isn't sufficient, try iterative DPO. Regenerate your preference data set from the current DPO model and run another round. Only if iterative DPO plateaus, should you escalate to PPO infrastructure. Seven takeaways to lock in. One, data set quality beats algorithm choice. Filter near ties, check the length bias correlation target below 0.2, D dub by prompt. A bad preference data set will fail no matter what algorithm you use. Two, the full Q Laura DPO setup for Llama 3.18B fits on a single RTX 4090. Four-bit policy, four-bit reference, Laura rank 64, learning rate 5e-7, beta 0.1, and our host alpha 1.0. Three, watch rewards/accuracies target above 70, rewards/margins growing, and KL divergence growing slowly. Stop training if your SFT evaluation benchmark drops more than three points. Four, use and our host alpha equals 1.0, the NLL auxiliary loss on chosen responses by default. It dramatically improves training stability and prevents the log PS/chosen collapse failure mode. Five, DPO achieves about 80% of PPO's alignment gains at 20 to 30% of the compute cost with better preservation of SFT capabilities. It's the right default for most alignment projects. Six, the escalation path is DPO, then iterative DPO, then PPO. Only go to PPO when you have online generation needs, verifiable reward signals, or iterative DPO has genuinely plateaued. Seven, evaluate at three tiers. Offline benchmarks like MT-Bench and AlpacaEval, AI judge win rate versus your SFT baseline, and capability regression checks every 500 steps. All three tell you different things. That's the complete DPO implementation from raw preference data to an evaluated aligned model. Here's your concrete action item for this week. Clone TRL's repository and run the DPO example script against the HHRLF dataset on whatever model fits your GPU. Watch rewards and accuracies climb in weights and biases. When it crosses 65%, generate 10 responses from both your SFT model and your DPO model on the same prompts and compare them qualitatively. You'll see the difference immediately. The DPO model will be noticeably less willing to produce low-quality responses and more consistently structured in its answers. Resources: TRL DPO config documentation, the accelerate FSDP configuration guide, Alpaca Eval for evaluation, and LM/evaluation harness. The TRL documentation specifically has worked examples for most common architectures. Like the video if it was useful. It genuinely helps the series reach engineers who need this. Subscribe for more and drop a comment with what alignment challenge you're working on. Many of my next topics come directly from your questions. See you in the next one.