When a user clicks "Submit", they shouldn't wait while your server sends three emails, generates a PDF and calls a slow third-party API. Those jobs belong in the background. In Django, the standard tool for that is Celery, usually with Redis as the message broker.
How it works
- Your Django view puts a task message on a queue (Redis) and responds immediately.
- One or more Celery workers, separate processes or containers, pick up tasks and run them.
- Celery Beat (optional) adds scheduled tasks, like cron for your app.
Minimal setup
# config/celery.py
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
# settings.py
CELERY_BROKER_URL = os.environ.get("REDIS_URL", "redis://localhost:6379/0")
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
# applications/tasks.py
from celery import shared_task
@shared_task(bind=True, autoretry_for=(ConnectionError,),
retry_backoff=True, max_retries=5)
def send_certificate_email(self, application_id: int):
application = Application.objects.get(pk=application_id)
if application.certificate_emailed:
return # idempotent: already done
send_email_with_pdf(application)
application.certificate_emailed = True
application.save(update_fields=["certificate_emailed"])
Rules for reliable tasks
Pass IDs, not objects
Send application_id, not the model instance. The task loads fresh data when it runs, and messages stay small and serialisable.
Make tasks idempotent
Tasks can run more than once (retries, worker restarts, acks_late). Design them so a second run does nothing harmful. Check a flag or use a unique constraint, as in the example above.
Queue after the transaction commits
If you queue a task inside a database transaction, the worker may start before the data is saved and fail to find it. Use:
from django.db import transaction
transaction.on_commit(lambda: send_certificate_email.delay(app.id))
Retry with backoff
Temporary failures (network errors, rate limits) should retry automatically with increasing delays. Permanent failures (invalid data) shouldn't retry forever. Log them and alert someone.
Set time limits
Use soft_time_limit and time_limit so one stuck task can't block a worker indefinitely.
Separate queues for different work
Route slow jobs (reports, AI calls, scraping) to their own queue and workers, so they never delay quick, user-facing ones like password-reset emails.
Scheduled jobs with Celery Beat
CELERY_BEAT_SCHEDULE = {
"expire-certificates-nightly": {
"task": "applications.tasks.expire_certificates",
"schedule": crontab(hour=1, minute=0),
},
"reconcile-payments": {
"task": "payments.tasks.reconcile_pending",
"schedule": crontab(minute="*/15"),
},
}
Run exactly one Beat process, otherwise scheduled tasks run multiple times.
Monitoring
Flower gives a live view of workers and tasks. In production, also export metrics (queue length, task failures, runtimes) to Prometheus and alert when queues grow or failures spike.
Deployment
Workers use the same Docker image as the web app with a different command, for example celery -A config worker -Q default,emails. On Kubernetes, run them as separate Deployments and scale them on queue length.
Lighter alternatives
For small projects, RQ, Django-Q2 or Dramatiq are simpler to run. Recent Django versions also introduced a built-in tasks framework for defining background work. Celery remains the most complete option for complex workloads with routing, scheduling and retries.
Need help making your Django app faster and more reliable? Our Python development team can help.