Efficient Preference Poisoning Attack on Offline RLHF
📰 ArXiv cs.AI
Learn how to efficiently poison offline RLHF pipelines using label flip attacks and understand the vulnerability of log-linear DPO to preference poisoning attacks
Action Steps
- Identify the vulnerability of log-linear DPO to label flip attacks
- Analyze the effect of flipping one preference label on the DPO gradient
- Develop a targeted poisoning attack using the key property of parameter-independent shift in the DPO gradient
- Evaluate the efficiency of the poisoning attack on offline RLHF pipelines
- Implement defenses against preference poisoning attacks in RLHF pipelines
Who Needs to Know This
AI researchers and engineers working on RLHF pipelines can benefit from understanding the potential vulnerabilities of their systems to preference poisoning attacks, and how to defend against them
Key Insight
💡 Log-linear DPO is vulnerable to label flip attacks, which can induce a parameter-independent shift in the DPO gradient
Share This
🚨 New attack on offline RLHF pipelines: efficient preference poisoning using label flip attacks 🚨
Key Takeaways
Learn how to efficiently poison offline RLHF pipelines using label flip attacks and understand the vulnerability of log-linear DPO to preference poisoning attacks
Full Article
Title: Efficient Preference Poisoning Attack on Offline RLHF
Abstract:
arXiv:2605.02495v1 Announce Type: cross Abstract: Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisonin
Abstract:
arXiv:2605.02495v1 Announce Type: cross Abstract: Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisonin
DeepCamp AI