Bounded Concurrency for LLM APIs in Python: Avoid 429s with HTTPX

Async LLM calls are not enough without bounded concurrency. Speed turns into 429s, retries, and wasted cost. You'll see how to connect [music] the pieces into a workflow that limits concurrent prompts and gathers [music] resilient responses. I'm Professor Pi, teaching AI engineering [music] and LLM systems with simple Python. If you fire hundreds of prompts at LLM provider without [music] a plan, you will hit rate limits and timeouts. Bounded concurrency is the practice of capping how many requests you let run at once. Success looks like steady throughput, fewer errors, and predictable [music] latency under load. For small teams that run experiment batches or production inference, that predictability [music] is where you save time and money. Think of bounded concurrency like traffic lights for requests. [music] Let too many cars go and the intersection [music] jams. Let the few through at a controlled pace and everything flows. The trade-off is simple. Cap too low and you add latency. [music] Cap too high and you invite 429s. We will show [music] how to find a sane cap using a mocked provider so you can test without burning credits. We will build a small experiment that warms a shared HTTP [music] client, simulates a burst that causes 429s, captures timing and failures, then applies a semaphore and aligned [music] connection limits to stop the storm. Quick note, this code is designed to teach the core idea clearly. Library versions and local setups can vary. So, check the official docs if your setup behaves differently. Now, we run a quick warm-up against a fake endpoint to confirm the client plumbing works before we stress it. This snippet primes an HTTPX async client against a fake LLM route so we can validate the plumbing >> [music]>> before we push load. The module level >> [music] >> fakes of the one completions response. We create [music] one shared client and warm-up awaits client.post with a tiny [music] high prompt to exercise the JSON path and response handling. Returning the completion ID proves the client actually hit the route rather than returning a cached object. And the printed [music] ID is your quick visual that the mock API is responding and the client is ready [music] for heavier tests. With the warm-up green, we deliberately remove [music] limits and simulate a burst to see what breaks. This example reproduces an uncontrolled [music] burst so you can see rate limits in action rather than guessing. Bounded LLM transport models a provider that sleeps 100 [music] milliseconds and starts returning 429 once more than three requests are active. Burst and reset closes the warm-up [music] client, recreates the transport, then fans out six simultaneous client.post calls with asyncio.gather to mimic a real [music] spike. Without any throttle, roughly half the calls arrive as 429s and the function returns [music] a failure count that exposes that naive parallelism already trips the limiter. Shown by burst rate limited equals three in the output. Now, we need to know which prompts [music] die first and how long the successful ones took. This code turns the failure storm into usable telemetry by timing every HTTPX completion under load. So, you can prioritize fixes. The nested hit co-routine timestamps each client.post, then returns the prompt text, HTTP status, and elapsed seconds. Capture metrics launches one co-routine per prompt with asyncio.gather, so you see exactly which responses [music] made it back while the transport was overloaded. Sorting the results in [music] place ranks the worst outcomes first, and the reported worst status equals 429, worst latency equals 0.10s points to the right problem [music] to address. Me can fix it by adding a simple gate that [music] never lets more than the safe limit run at once. This snippet adds an from blasting [music] the transport beyond its safe parallelism, which is all you really need to stop a 429 [music] storm. The helper bounded run creates a gate sized to the provider [music] limit, and wraps each client.post inside async with gate, while still using asyncio.gather, so queue tasks [music] drain as the service allows. With the gate set to three, the mocked limiter stops issuing [music] 429s, and every co-routine completes cleanly. Bump the semaphore above three, and you'll reintroduce spikes. Lower it, and you trade [music] throughput for extra safety. The console reading bounded_429s [music] equals zero confirms the guardrail is working. To make the whole stack consistent, align [music] the client pool limits with the semaphore, so sockets and tasks agree on the cap. This example finalizes the pattern by aligning HTTPX limits with the semaphore so the entire stack respects the same ceiling. We shut down the previous client, rehydrate it with HTTPX.limits max connections equals three max keep alive connections equals three and reuse the shared transport so pool sizing and server back pressure stay in sync. The global gate mirrors that cap while safe batch measures each guarded client.post to produce a throughput and latency summary. With pulled keep alive connections, a nine call [music] burst stays inside the allowed lanes without opening extra sockets or reintroducing contention. The reported values [music] bounded success equals nine average latency equals 0.10s show a healthy well-bounded batch [music] you can confidently scale from. This toy flow maps directly into a real project. Replace the mock transport with your provider, tune the gate to the provider guidance, and run the warm up and metrics [music] capture before going live. The warm up proves credentials and JSON work. Metrics tell you [music] where to tune and the aligned pool plus semaphore gains predictable safe [music] throughput. Recap. Bounded concurrency is simply a cap on in-flight requests so providers stop returning 429s. Use it [music] anytime you run batches, streaming, or many parallel inferences and need reliable [music] latency. The catch is you must pick the cap from observed latency [music] and your provider limits. Too low adds unnecessary delay. Too high invites [music] rate limits. Next step, tune the semaphore and max connections together to find the point that maximizes throughput [music] while keeping errors at zero. If short, practical AI engineering helps, subscribe and watch the AI engineering videos.