DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression

📰 ArXiv cs.AI

Learn how to apply Diagonal-Anchored Repulsive Knowledge Distillation (DARK) for vision-language models to achieve better compression without sacrificing performance

advanced Published 9 May 2026

Action Steps

Apply knowledge distillation to vision-language models using the DARK method
Use diagonal-anchored repulsive loss to reduce architectural biases
Evaluate the performance of the compressed model under extreme compression
Compare the results with traditional knowledge distillation methods
Fine-tune the hyperparameters to optimize the compression ratio and performance

Who Needs to Know This

Computer vision and natural language processing teams can benefit from this technique to deploy models on devices with limited resources, improving performance in clinical settings

Key Insight

💡 DARK helps to preserve the pairwise similarity structure of the teacher model, even under extreme compression, by reducing architectural biases

Key Takeaways

Learn how to apply Diagonal-Anchored Repulsive Knowledge Distillation (DARK) for vision-language models to achieve better compression without sacrificing performance

Full Article

Title: DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression

Abstract:
arXiv:2603.05421v3 Announce Type: replace-cross Abstract: Compressing vision-language models for on-device deployment is increasingly important in clinical settings, but knowledge distillation (KD) degrades sharply when the teacher-student capacity gap spans an order of magnitude or more. We argue that, under such gaps, strict imitation of the teacher is a poor objective: much of the teacher's pairwise similarity structure reflects its own architectural biases rather than information a compact s

Read full paper → ← Back to Reads