You Probably Don't Need Celery in Your FastAPI App
A lot of FastAPI developers end up with Celery not because they need a distributed task queue, but because BackgroundTasks stopped being enough and Celery was the first thing that came up when they searched for a solution.
This is about that gap, and a library that fills it without the overhead.
What BackgroundTasks does not give you
FastAPI's built-in BackgroundTasks is straightforward. You attach a function to the response and Starlette calls it after the response is sent.
@app.post("/signup")
def signup(email: str, background_tasks: BackgroundTasks):
background_tasks.add_task(send_welcome_email, email)
return {"ok": True}
That covers fire-and-forget. But in a real application you quickly hit walls:
No retries. If send_welcome_email fails because the SMTP server returned a 503, the task is gone. There is no retry, no backoff, no record of what happened.
No persistence. If the app restarts during a deploy, every queued task disappears. Tasks that were waiting to run simply never run.
No visibility. You cannot see what is running, what has run, what failed, or how long things took. The only way to know a task failed is to catch it in your logs, if you are logging at all.
No scheduling. BackgroundTasks runs things once, after the current request. There is no built-in way to run something on a schedule.
These are not edge cases. They are the baseline requirements for any background job in production.
Why Celery is the wrong answer for most of these
When developers hit these limitations, the standard advice is: add Celery. And Celery does solve all four problems. But it solves them by giving you a distributed task queue, which comes with the full infrastructure that entails.
To use Celery with FastAPI you need:
- A message broker. Usually Redis or RabbitMQ. A separate service to run, configure, monitor, and back up.
- A worker process. A separate process that consumes from the broker. Needs to be deployed, restarted on failure, and kept in sync with the app on every deploy.
- Celery Beat for scheduling. Another separate process.
- Flower or similar if you want visibility. Yet another service.
Celery was built for teams running tasks on dedicated workers across multiple machines at high volume. If that describes your situation, it is the right tool. But most FastAPI apps sending emails, processing uploads, running nightly reports, and syncing data are not in that category. They just needed BackgroundTasks to grow up a little.
What actually fills the gap
fastapi-taskflow is built specifically for this problem: FastAPI apps that have outgrown BackgroundTasks but do not need a distributed task queue.
It runs inside your FastAPI process. No broker. No separate worker. Tasks execute the same way they do today, after the response, but now with retries, persistence, scheduling, and a live dashboard.
Setup:
from fastapi import BackgroundTasks, FastAPI
from fastapi_taskflow import TaskAdmin, TaskManager
task_manager = TaskManager(snapshot_db="tasks.db", requeue_pending=True)
app = FastAPI()
TaskAdmin(app, task_manager, auto_install=True)
Retries:
@task_manager.task(retries=3, delay=60.0, backoff=2.0)
def send_welcome_email(email: str):
_send(email) # raise any exception — the retry handles it
The function stays a plain function. Raise an exception on failure and it retries automatically with exponential backoff.
Your routes stay the same:
@app.post("/signup")
def signup(email: str, background_tasks: BackgroundTasks):
background_tasks.add_task(send_welcome_email, email=email)
return {"ok": True}
Same annotation. Same calling convention. Nothing changes in your routes.
Persistence across restarts:
requeue_pending=True saves tasks that were queued at shutdown and re-dispatches them on the next startup. Tasks no longer disappear on deploy.
Scheduling:
@task_manager.schedule(cron="0 2 * * *")
def nightly_cleanup():
_run_cleanup()
No Beat process. No separate service. The scheduler runs inside the app.
Eager dispatch:
BackgroundTasks always runs after the response is sent. If you need a task to start immediately, before the response goes out, set eager=True:
@task_manager.task(retries=3, eager=True)
async def notify_user(user_id: int):
await push_service.send(user_id, "Your request is processing")
The task starts via asyncio.create_task the moment add_task() is called. It is still tracked, still retried on failure, still visible in the dashboard. You can also set it per call:
background_tasks.add_task(notify_user, user_id, eager=True)
Visibility:
/tasks/dashboard is a live dashboard that shows every task, its current status, duration, logs, and the full stack trace on failure. It updates over SSE in real time. No Flower setup, no external monitoring service.
The honest trade-offs
This is not a Celery replacement. If your tasks are CPU-intensive and need isolation from request handlers, if you need to route different task types to dedicated worker machines, or if you are processing thousands of tasks per minute, you need a proper task queue.
What fastapi-taskflow covers is the case where you reached for Celery because BackgroundTasks gave you nothing, not because you genuinely needed distributed workers.
For a single-host deployment, multiple instances on the same host share a SQLite file. For multiple hosts, swap to Redis or PostgreSQL as the backend and idempotency, requeue claiming, and task history all work across instances without any coordination overhead.
What you skip entirely
No broker to run or monitor. No worker process to deploy or restart. No Celery app instance or separate tasks module. No Beat process for scheduling. No Flower for visibility.
Local development stays at uvicorn app.main:app. New developers on the project do not need to learn a separate system.
The four things that pushed you toward Celery in the first place, retries, persistence, scheduling, and visibility, are covered.
