Understand the four metric types, then scrape real machine metrics with node_exporter — CPU, memory, disk — the data behind most infrastructure dashboards.
Why: every Prometheus metric is one of four types, and knowing which is which decides how you query it. A counter only goes up (requests served) — you query its rate. A gauge goes up and down (memory in use, temperature) — you read it directly. A histogram buckets observations (request durations) for percentiles. A summary is similar but computes quantiles client-side.
counter ─ only increases; reset to 0 on restart → use rate()
gauge ─ goes up and down → read directly
histogram ─ buckets of observations (e.g. latency) → histogram_quantile()
summary ─ client-side quantiles → read the quantileWhy: a target exposes metrics as plain text at /metrics — one metric per line, with optional labels in {braces} and a value. # HELP and # TYPE lines describe each metric. Prometheus parses exactly this format. Labels (like method and status) let you slice one metric many ways.
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1027
http_requests_total{method="POST",status="500"} 3
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5.243e+07Why: Prometheus does not collect OS metrics itself — an "exporter" does, exposing them at /metrics for Prometheus to scrape. node_exporter is the standard one for host metrics: CPU, memory, disk, network. Add it to your Compose stack and it starts publishing hundreds of node_* metrics.
# add to docker-compose.yml services:
node_exporter:
image: prom/node-exporter:latest
ports: ["9100:9100"]Why: a new exporter is useless until Prometheus scrapes it. Add a job pointing at node_exporter:9100 (the service name resolves on the Compose network), then reload. The Targets page should show the new job UP, and node_* metrics become queryable.
# add to prometheus.yml under scrape_configs:
- job_name: node
static_configs:
- targets: ["node_exporter:9100"]Why: Prometheus can reload its config without a restart if started with --web.enable-lifecycle, or you restart the container. Then query the node metrics. These three are the backbone of any host dashboard — available memory, filesystem space, and CPU time by mode.
Apply the new config (restart is the simplest)
docker compose restart prometheusThen in the expression browser, try: node_memory_MemAvailable_bytes node_filesystem_avail_bytes node_cpu_seconds_total