LLM Batch Inference in Python with Ray Data: Run Large Eval Jobs Faster

Big eval jobs should not be a slow for loop. Ray data lets you scale LLM batch inference across data sets, engines, and hosted endpoints. This is the setup you'll be able to run after this, which labels many texts in parallel and reports job throughput. I'm Professor Pi, teaching AI engineering and LLM systems with simple Python. At scale, you can't treat evaluation as a bunch of one-off scripts. You'll hit overhead from one process per prompt, from extra network roundtrips, and from Python call costs. The practical goal is labeling thousands or millions of examples reliably while measuring both accuracy and throughput. Ray data fits because it treats your examples as a distributed data set and runs transforms in parallel across workers. So you get predictable batching, straightforward aggregation, and fewer surprises than handrolled multipprocessing. Think of Ray data like a spreadsheet you can ship out to many workers. You write small functions that transform rows or batches. Ray splits the sheet up, runs those functions in parallel, and stitches the results back together. The tradeoff is learning when to use map versus map batches and tuning batch size so latency and memory stay in balance. We'll build a tiny evaluation loop, create a small labelled data set, run a mock model, compute errors in batches, inspect worst cases, try a quick normalization trick, and then show batched HTTP calls that keeps everything concrete. Raw text to metrics so you can take the pattern to bigger jobs. Let's start with a tiny example you can run locally. This snippet creates a tiny ray data data set and attaches a mock LLM prediction to each row using data set.map. It first defines small labeled text pairs to mimic prompts and target sentiments. The function call LLM is a deterministic standin for model output mapping obvious keywords to simple labels. The data set is constructed with ray data from items. So each record holds an ID, the text, and the ground truth label. A map operation adds a PR field by applying call LLM to the text column. Finally, it counts rows and peaks at the first prediction to show the transform worked. The printed values confirm data loaded and predictions were produced. Now we'll batch the evaluation so we reduce per call overhead. This code evaluates predictions in batches using ray data's data set matt batches to add an error flag per row. The function eval batch receives a list of records and returns the same records with an error field set to one when pred differs from label. The call to map batches applies that logic over batches of size two, illustrating how to process multiple rows per function invocation. Next, data sets sum aggregates total errors and data set.count returns the population size. The final accuracy is computed as one minus error rate and rounded for readability. Increasing batch size reduces Python call overhead but may raise memory usage. Smaller values improve fairness under variable payloads. The printed accuracy quantifies baseline quality. Let's look at the mistakes so we know what to fix first. This example pinpoints difficult cases by filtering, enriching, and ranking with Ray data. It first filters rows to those where the air flag equals 1 using data set.filter yielding only mismatches. Then it maps a small helper that adds a chars field with the text length giving a quick proxy for prompt complexity. The data set is sorted by chars in descending order via data set.sort to bring the most verbose failures to the top. A single take returns the worst offender for inspection. The printed ID and character count identify a concrete failing example guiding where to focus prompt or model fixes. A light pre-processing step often buys noticeable gains. Let's try one. This snippet improves results by normalizing text before inference and recomputing metrics with ray data. A normalize function rewrites the phrase not bad to good and lowercases input aligning with the mock models keyword logic using data set.map. It adds pred 2 by running call llm on the normalized text. Then maps again to attach error 2 for correctness tracking. The count and sum operations compute how many examples were processed and how many remained wrong after the tweak. If you expand normalization to handle more patterns, accuracy can rise further. If it is too aggressive, it may mask genuinely negative inputs. The printed tuned accuracy shows the benefit of pre-processing. Finally, here's how to wire up batched HTTP calls without exploding your control flow. This code shows how to batch HTTP inference inside ray data using data set.mmap batches with a requests.post call. The function postb submits each text as a prompt to a local openi style endpoint and parses the JSON response to extract a choice text. If the endpoint is unavailable, it falls back to call LLM, preserving the batch jobs reliability. The batch size parameter controls how many records are processed per worker invocation, trading request overhead against memory and downstream rate limits. After transformation, take all gathers results to compute how many unique predictions were produced. The printed count confirms end-to-end wiring and helps validate that batched calls return varied outputs. Taken together, this toy loop mirrors production checks. The same pattern scales to large corpora. Swap the mock model for a hosted engine. Add robust retry and error handling and you've got a resilient bulk inference pipeline. Recap. Ray data is a distributed data set layer that runs maps and batch maps across workers. Use it when you need to label or evaluate large sets of prompts and want predictable batching plus easy aggregation. One caveat, batch size and pre-processing both affect memory use and the signals your model sees, so tune them carefully. Next step, increase the batch size in map batches and measure throughput and memory to find the best trade-off for your endpoint and worker size. If short practical AI engineering helps, subscribe and watch the AI engineering