Make Prometheus work for you in the background — precompute expensive queries with recording rules, and fire alerts when conditions hold for long enough with alerting rules.
Why: Prometheus evaluates rules on a schedule. A recording rule precomputes a query and saves the result as a new metric — so dashboards read a cheap precomputed series instead of a heavy query every refresh. An alerting rule evaluates a condition and fires an alert when it is true. Both live in rule files Prometheus loads.
recording rule ─ run a query on a schedule, store the result as a metric
(makes dashboards fast; standardizes common expressions)
alerting rule ─ run a condition; if true for "for:" duration → fire alert
(sends to Alertmanager)Why: if every dashboard recomputes the same error-rate expression, you pay for it repeatedly. A recording rule computes it once per interval and stores it under a clean name (the convention is level:metric:operation). Dashboards then query the short name.
# rules.yml
groups:
- name: app_rules
interval: 30s
rules:
- record: job:http_requests:rate5m
expr: sum by (job) (rate(http_requests_total[5m]))Why: an alerting rule fires when its expr is true continuously for the for: duration — the "for" prevents flapping on a brief blip. labels set severity for routing; annotations carry the human-readable message. This one alerts when the 5xx error ratio exceeds 5% for 10 minutes.
# rules.yml (add another group)
- name: app_alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: critical
annotations:
summary: "High 5xx error rate on {{ $labels.job }}"Why: Prometheus only evaluates rule files it is told about. Point rule_files at your file in prometheus.yml, then reload. The Alerts page in the UI shows each alert as inactive, pending (condition true, waiting out the for:), or firing.
# prometheus.yml
rule_files:
- rules.yml
# (then: docker compose restart prometheus
# and watch Status → Rules and the Alerts page)