Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility

📰 ArXiv cs.AI

Learn how to optimize long-context inference in decoder-only language models using the Shallow Prefill, Deep Decoding technique

advanced Published 9 May 2026

Action Steps

Apply the Shallow Prefill, Deep Decoding technique to reduce computational costs
Implement layer-asymmetric KV visibility to materialize non-anchor prompt-token KV states only in lower layers
Configure the model to keep Decode-phase tokens full-depth
Test the optimized model on long-context inference tasks
Compare the performance of the optimized model with the original model

Who Needs to Know This

NLP engineers and researchers can benefit from this technique to improve the efficiency of their language models, especially when dealing with long prompts

Key Insight

💡 Shallow Prefill, Deep Decoding can significantly reduce computational costs in long-context inference tasks

Full Article

Title: Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility

Abstract:
arXiv:2605.06105v1 Announce Type: new Abstract: Long-context inference in decoder-only language models is costly because long prompts are processed during Prefill, cached at every layer, and repeatedly attended to during autoregressive Decode. We introduce \emph{Shallow Prefill, dEEp Decode} (SPEED), a phase-asymmetric KV-visibility policy that materializes non-anchor prompt-token KV states only in lower layers while keeping Decode-phase tokens full-depth. Unlike previous approaches that make up

Read full paper → ← Back to Reads