Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints

📰 ArXiv cs.AI

Researchers propose a batch-level routing framework for large language models to optimize query routing under cost and capacity constraints

advanced Published 31 Mar 2026

Action Steps

Identify cost and capacity constraints for large language models
Develop a batch-level routing framework to optimize model assignment
Implement resource-aware routing to respect cost and model capacity limits
Evaluate the framework's performance under non-uniform or adversarial batching

Who Needs to Know This

This research benefits data scientists, AI engineers, and DevOps teams working with large language models, as it helps optimize resource utilization and reduce costs

Key Insight

💡 Batch-level routing can help control costs and optimize resource utilization for large language models

Key Takeaways

Researchers propose a batch-level routing framework for large language models to optimize query routing under cost and capacity constraints

Full Article

Title: Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints

Abstract:
arXiv:2603.26796v1 Announce Type: cross Abstract: We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch while respecting cost and model capacity limits. We further intro

Read full paper → ← Back to Reads