Online Linear Programming for Multi-Objective Routing in LLM Serving

📰 ArXiv cs.AI

Learn to optimize multi-objective routing in LLM serving using online linear programming for better latency-throughput trade-offs

advanced Published 7 Jul 2026

Action Steps

Formulate the routing problem as an online linear program using constraints such as batch-size and KV-cache limits
Implement a multi-objective optimization framework to balance latency and throughput
Solve the online linear program using a suitable algorithm to obtain optimal routing decisions
Test and evaluate the performance of the optimized routing strategy using metrics such as latency and throughput
Compare the results with existing routing heuristics to demonstrate the improvement

Who Needs to Know This

This benefits devops and software engineers working on large language model serving, as it provides a framework for optimizing routing decisions under tight constraints

Key Insight

💡 Online linear programming can be used to optimize multi-objective routing in LLM serving, providing better control over latency-throughput trade-offs

Key Takeaways

Learn to optimize multi-objective routing in LLM serving using online linear programming for better latency-throughput trade-offs

Full Article

Title: Online Linear Programming for Multi-Objective Routing in LLM Serving

Abstract:
arXiv:2607.03948v1 Announce Type: new Abstract: We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-level objectives (SLOs) and offer limited control over latency-throughput trade-offs, we introduce a multi-objective optimization framework that formulates routing as an onl

Read full paper → ← Back to Reads