Every production system eventually breaks. What matters is whether you notice first or your users do. Prometheus and Grafana are the most widely used open-source monitoring stack, and a useful starter setup takes days, not months.
The pieces
- Prometheus collects ("scrapes") numeric metrics from your services and infrastructure at regular intervals and stores them as time series.
- Exporters expose metrics for things that don't do so natively: node-exporter for servers, postgres-exporter, redis-exporter, blackbox-exporter for uptime checks.
- Alertmanager routes alerts to email, Slack, Microsoft Teams, PagerDuty and others, with grouping and silencing.
- Grafana turns metrics into dashboards, and can also show logs (Loki) and traces (Tempo).
On Kubernetes, the kube-prometheus-stack Helm chart installs all of this with sensible default dashboards and alerts in one step.
What to measure first
For every service: the four golden signals
- Latency: how long requests take (track the 95th/99th percentile, not just the average),
- Traffic: requests per second,
- Errors: the rate of 5xx responses and failed jobs,
- Saturation: how full things are, such as CPU, memory, queue length and database connections.
For Django and Python apps
django-prometheus or the official prometheus_client library expose request counts, latencies and database metrics with a few lines of configuration. Add custom business metrics too, because they're often the earliest warning sign:
from prometheus_client import Counter
payments_confirmed = Counter(
"payments_confirmed_total", "Payments confirmed by gateway callback")
# in your callback handler
payments_confirmed.inc()
A sudden drop in confirmed payments can reveal a broken gateway integration before any error appears.
Alerts that matter
Start with a small set of symptom-based alerts, the ones users would feel:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 10m
labels: { severity: page }
annotations:
summary: "More than 5% of requests are failing"
Other good first alerts: site down (blackbox probe failing), latency far above normal, disk more than 85% full, certificate expiring within 14 days, background queue growing continuously, and pods crash-looping.
Avoid alert fatigue
- Every alert must be actionable. If nobody needs to do anything, it's a dashboard item, not an alert.
- Use
for:durations so brief spikes don't page anyone. - Separate "page now" from "look tomorrow" severities.
- Link each alert to a short runbook: what it means and the first steps to take.
- Review noisy alerts monthly and fix or delete them.
Dashboards people actually use
Build one overview dashboard per system showing the golden signals and key business metrics on a single screen, and link to detailed dashboards from there. Import community dashboards for standard components (nodes, PostgreSQL, Redis, NGINX) instead of building them from scratch.
Next steps
Once metrics are in place, add centralised logs (Loki or the ELK stack) and tracing with OpenTelemetry to follow a single request across services. Grafana can bring all three together, so you can jump from a spike on a graph to the exact logs and traces behind it.
Want monitoring that catches problems before your users do? Our DevOps team can set it up and tune it for your stack.