NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
📰 ArXiv cs.AI
Learn how NetKV improves LLM inference by selecting decode instances based on network awareness, reducing Time to First Token (TTFT) budget
Action Steps
- Implement a network cost oracle to estimate transfer times between prefill and decode instances
- Use the network cost oracle to inform the scheduler about topological distance and dynamic congestion
- Configure the scheduler to route requests based on both compute load and network awareness
- Test the NetKV approach with disaggregated LLM inference workloads
- Compare the performance of NetKV with existing schedulers in terms of TTFT budget
Who Needs to Know This
ML engineers and researchers working on LLM inference can benefit from this knowledge to optimize their models' performance, especially in distributed computing environments
Key Insight
💡 Network awareness is crucial in optimizing LLM inference performance, especially in distributed computing environments
Share This
🚀 NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference 🚀
Key Takeaways
Learn how NetKV improves LLM inference by selecting decode instances based on network awareness, reducing Time to First Token (TTFT) budget
Full Article
Title: NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
Abstract:
arXiv:2606.03910v1 Announce Type: cross Abstract: Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget. Current schedulers route on compute load and prefix-cache locality alone, ignoring the topological distance and dynamic congestion between prefill and decode instances. We close this gap with a thin operator-to-scheduler interface, the network cost oracle, and we pr
Abstract:
arXiv:2606.03910v1 Announce Type: cross Abstract: Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget. Current schedulers route on compute load and prefix-cache locality alone, ignoring the topological distance and dynamic congestion between prefill and decode instances. We close this gap with a thin operator-to-scheduler interface, the network cost oracle, and we pr
DeepCamp AI