FastAPI BackgroundTasks vs. Celery: Which One to Choose?

Key Takeaways

Hey, welcome back. So far, every endpoint we've built returns a response only after all the work is done. That's fine for a quick database query or a short LLM call. But what about document ingestion? Chunking a 50page PDF. Generating embeddings for every chunk and indexing everything into QR can take 10 to 30 seconds. Holding the HTTP connection open for that long is a bad experience. In this video, we cover background tasks. How to return a response immediately and continue the work after. Think about what happens when a user uploads a 100page PDF to Karan's RA API. The endpoint has to read the file, split it into chunks, call the embedding API for each chunk, and index everything into QR. That's easily 20 to 30 seconds. During all of that, the HTTP connection is open and the caller is sitting there waiting. Most proxies and API gateways have a 30 to 60-second timeout. If the file is large enough, the connection drops before the work is done and the caller gets an error even though everything succeeded. The right pattern is to accept the job, return a response immediately and do the heavy work in the background. The caller gets instant feedback and the work still gets done. The difference is one parameter and one line. Look at the two versions. Without background tasks, the endpoint does all the slow work before returning. The caller waits the full 30 seconds. With background task, the endpoint reads the file, registers the slow work as a background task, and immediately returns a response. The caller gets their response right away. Fast API then runs process document after the response is sent using the same server process. The caller sees processing instantly. The heavy work happens behind the scenes. That's the core pattern. Fast API's background tasks is as simple as it gets. Declare it as a parameter in your endpoint and fast API injects it automatically. Call background_task add task with the function you want to run and any arguments it needs. Notice the status code is 202 accepted not 20 okay that's the correct HTTP status for I received your request and I'm working on it. The process document function runs after the response is sent. It gets the file content, the user ID, and the vector store client. Everything it needs to do the full ingestion without holding the connection open. Background task is great for lightweight post response work. Sending a notification, updating a counter, logging something, but it has real limits. If your server crashes while a task is running, the task is gone. There's no retry mechanism. if the embedding API fails halfway through and there's no way to check whether a task succeeded or failed. For heavy ingestion jobs, longunning agent workflows, or anything that needs to survive a server restart, you need a proper job queue. Celery with Reddis is the most common Python choice. The job Q stores the task persistently, retries on failure, and lets you track job status. Use background tasks for simple fire and forget work. Use a job queue when durability and visibility matter. The job pattern is the professional version of background processing. Instead of just returning processing, you return a job ID. The caller stores that ID and pulls a separate status endpoint to check on progress. Look at the two endpoints. Post/document/ingest creates a job, stores its status as pending, kicks off the background work, and returns the job ID immediately. Get/job/openrace job close brace lets the caller check status at any time. The background task updates the job status as it progresses from pending to processing to completed or failed. One important note, the in-memory jobs dictionary works for demonstration but not production. In production, store job state in Reddis or a database so it survives server restarts. Quick recap. Never hold a slow HTTP connection open. Return immediately and continue work in the background. Fast API's background tasks handle simple post response work with no extra setup. For heavy longunning jobs, use a proper queue like celery with readis. And when callers need to track progress, use the job pattern. Return a job ID immediately and let them pull a status endpoint. Next up is logging and observability. How to trace what's happening inside Karan's API, catch provider errors, and keep visibility into your system in production. See you there.