The default Prometheus install keeps 15 days of metrics on local disk. Past that, the question of “where do older metrics live, at what resolution, and how do we query across them” is one teams either solve deliberately or discover during an audit that requires a year of data they no longer have. This article describes the retention policy pattern we apply for clients — a hot tier for fast queries, a warm tier for long-term and downsampled data, and a hard disk cap on the hot tier so a scrape spike does not write the cluster into read-only mode.
How to verify
The questions an operator should be able to answer in seconds, on any environment:
# What retention is Prometheus actually applying?
curl -fsS http://prom:9090/api/v1/status/runtimeinfo | jq '{storageRetention, GOMAXPROCS}'
ls -la /var/lib/prometheus/wal | head
du -sh /var/lib/prometheus/{wal,*-*}
# What does the long-term store hold?
curl -fsS http://thanos-store:10902/loaded | jq '.blocks | length'
curl -fsS 'http://thanos-query:10902/api/v1/query?query=up&time=2025-01-01T00:00:00Z' | jq '.data.result | length'
# Compactor health (Thanos)
curl -fsS http://thanos-compactor:10902/metrics | grep -E '^thanos_compact_(group_compactions_total|group_failures_total|halted)'
A non-zero thanos_compact_halted means the compactor has stopped — usually because two Prometheus replicas are pushing overlapping blocks with different external labels. Past 24 hours of halt and you stop getting downsampling, then queries start returning incomplete data.
What’s happening
A Prometheus TSDB has two retention knobs: --storage.tsdb.retention.time (the maximum age of samples) and --storage.tsdb.retention.size (the maximum size of the on-disk database). Whichever triggers first wins. Beyond what the local TSDB holds, you need an external store — Thanos, Cortex/Mimir, VictoriaMetrics, or a managed service — which Prometheus writes to either through the remote_write protocol (push) or by exposing the local TSDB blocks via a sidecar (Thanos pattern).
Resolution matters as much as duration. Raw 15-second samples are useful for the last few weeks; for a year of data, downsampling to one-minute and one-hour summaries cuts the storage by 4×–240× while preserving the trends an audit or capacity-planning exercise cares about. Most long-term stores do this automatically — Thanos via thanos compact, Mimir via the compactor service, VictoriaMetrics via downsampling rules.
The three-tier shape we use:
- Hot (Prometheus local) — 15 days raw at 15s resolution, on fast local disk, sized for the worst plausible scrape spike plus headroom.
- Warm (long-term store) — 90–365 days, with raw samples for the first 30 days and downsampled (5m, 1h) past that.
- Cold (audit archive) — older than the warm tier, in object storage, queried infrequently via an on-demand store.
The procedure
-
Set hard retention bounds on every Prometheus. Both flags.
ExecStart=/usr/local/bin/prometheus \ --storage.tsdb.path=/var/lib/prometheus \ --storage.tsdb.retention.time=15d \ --storage.tsdb.retention.size=80GB \ ...The size cap is the safety net — if a misconfigured scrape suddenly emits 10× more series, the disk does not fill. Pair with an alert on
prometheus_tsdb_storage_blocks_bytes > 70 * 1024^3to give the operator runway before the cap kicks in. -
Pick a long-term store and wire
remote_write(or run a Thanos sidecar for the block-shipping pattern). Mimir/Cortex example:# /etc/prometheus/prometheus.yml remote_write: - url: https://mimir.internal/api/v1/push basic_auth: username: tenant-acme password_file: /etc/prometheus/mimir-token write_relabel_configs: - source_labels: [__name__] regex: 'go_gc_.*|process_open_fds|prometheus_tsdb_.*' action: drop queue_config: max_samples_per_send: 8000 capacity: 100000 min_backoff: 30ms max_backoff: 5sThe
write_relabel_configsdrops local-only series (Prometheus internal metrics, low-value Go runtime metrics) before they leave the box — a 30–50 % reduction in remote-write volume is common. -
Configure the long-term store’s retention and downsampling. Thanos example via the compactor:
thanos compact \ --data-dir=/var/lib/thanos-compactor \ --objstore.config-file=/etc/thanos/objstore.yml \ --retention.resolution-raw=30d \ --retention.resolution-5m=180d \ --retention.resolution-1h=2y \ --compact.concurrency=1Reading top-to-bottom: raw samples for 30 days, 5-minute downsamples for 6 months, 1-hour downsamples for 2 years. The
retention.resolution-*flags delete blocks older than the cutoff at that resolution, not earlier — the data is still queryable at the coarser resolution. -
Make sure the query layer can read across tiers transparently. Thanos Query / Mimir Query Frontend / VictoriaMetrics vmselect all federate Prometheus + long-term store; configure Grafana against the federated endpoint, not against Prometheus directly, so a query that goes past 15 days transparently hits the long-term store.
-
Cap cardinality at the source. Retention math assumes a stable series count. A label change that doubles cardinality doubles the storage of the entire retention window, retroactively. Add a Prometheus rule that alerts on rapid cardinality growth:
- alert: SeriesCardinalitySpike expr: | sum(prometheus_tsdb_head_series) / sum(prometheus_tsdb_head_series offset 1h) > 1.3 for: 15m labels: { severity: warning, team: platform } annotations: summary: 'Series count jumped 30%+ in the last hour' -
Document the retention policy and put it where auditors can find it. Three columns in a wiki page: tier (hot/warm/cold), retention (days/months), resolution (15s/5m/1h). For a regulated environment (PCI, HIPAA, SOC 2), the retention has to match what the auditor expects; missing this is one of the most common audit findings.
Operational notes
- Local Prometheus retention does not delete data the moment the threshold passes — block compaction runs at intervals and deletes the next compaction cycle. Expect 1–2 hours lag.
- Reducing the local retention does not free disk until the next compaction; do not panic when du does not drop immediately.
remote_writeblocks Prometheus’s main loop when the queue is full; an undersizedcapacitycauses scrape lag visible as missing samples in dashboards.- Thanos’s compactor must run for one tenant/owner per object-store prefix — two compactors on the same bucket racing each other will corrupt blocks.
- Audit retention requirements often specify “X years” and a sampling rate; without downsampling, “5 years at 15s” is petabytes per service. Downsampling makes the requirement realistic.
For the Prometheus install that sits at the hot tier, see prometheus-install-ubuntu. For an alternative store optimized for object storage, see grafana-mimir-cluster-s3-deploy.
Stack Harbor designs metrics retention against the audit window and the disk budget jointly, as part of the managed operations tier — hot/warm/cold tiers, downsampling explicitly tied to the resolution operators actually query, and cardinality alerted on before a label change doubles the entire archive.