Python & AI Automation

Background Jobs in Django with Celery and Redis

Home Blog Python & AI Automation

Python & AI Automation

When a user clicks "Submit", they shouldn't wait while your server sends three emails, generates a PDF and calls a slow third-party API. Those jobs belong in the background. In Django, the standard tool for that is Celery, usually with Redis as the message broker.

How it works

  1. Your Django view puts a task message on a queue (Redis) and responds immediately.
  2. One or more Celery workers, separate processes or containers, pick up tasks and run them.
  3. Celery Beat (optional) adds scheduled tasks, like cron for your app.

Minimal setup

# config/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

# settings.py
CELERY_BROKER_URL = os.environ.get("REDIS_URL", "redis://localhost:6379/0")
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
# applications/tasks.py
from celery import shared_task

@shared_task(bind=True, autoretry_for=(ConnectionError,),
             retry_backoff=True, max_retries=5)
def send_certificate_email(self, application_id: int):
    application = Application.objects.get(pk=application_id)
    if application.certificate_emailed:
        return                       # idempotent: already done
    send_email_with_pdf(application)
    application.certificate_emailed = True
    application.save(update_fields=["certificate_emailed"])

Rules for reliable tasks

Pass IDs, not objects

Send application_id, not the model instance. The task loads fresh data when it runs, and messages stay small and serialisable.

Make tasks idempotent

Tasks can run more than once (retries, worker restarts, acks_late). Design them so a second run does nothing harmful. Check a flag or use a unique constraint, as in the example above.

Queue after the transaction commits

If you queue a task inside a database transaction, the worker may start before the data is saved and fail to find it. Use:

from django.db import transaction
transaction.on_commit(lambda: send_certificate_email.delay(app.id))

Retry with backoff

Temporary failures (network errors, rate limits) should retry automatically with increasing delays. Permanent failures (invalid data) shouldn't retry forever. Log them and alert someone.

Set time limits

Use soft_time_limit and time_limit so one stuck task can't block a worker indefinitely.

Separate queues for different work

Route slow jobs (reports, AI calls, scraping) to their own queue and workers, so they never delay quick, user-facing ones like password-reset emails.

Scheduled jobs with Celery Beat

CELERY_BEAT_SCHEDULE = {
    "expire-certificates-nightly": {
        "task": "applications.tasks.expire_certificates",
        "schedule": crontab(hour=1, minute=0),
    },
    "reconcile-payments": {
        "task": "payments.tasks.reconcile_pending",
        "schedule": crontab(minute="*/15"),
    },
}

Run exactly one Beat process, otherwise scheduled tasks run multiple times.

Monitoring

Flower gives a live view of workers and tasks. In production, also export metrics (queue length, task failures, runtimes) to Prometheus and alert when queues grow or failures spike.

Deployment

Workers use the same Docker image as the web app with a different command, for example celery -A config worker -Q default,emails. On Kubernetes, run them as separate Deployments and scale them on queue length.

Lighter alternatives

For small projects, RQ, Django-Q2 or Dramatiq are simpler to run. Recent Django versions also introduced a built-in tasks framework for defining background work. Celery remains the most complete option for complex workloads with routing, scheduling and retries.

Need help making your Django app faster and more reliable? Our Python development team can help.