Metrics retention and storage

Topic: Monitoring basics

Set retention for metrics based on storage and query needs. Long retention uses more storage; downsample or archive for cost. Use when configuring or scaling a metrics system.

Intent: How-to

Quick answer

  • Keep raw metrics for short period; aggregate or downsample for longer. Example: 15s raw 7 days; 1m 30 days; 1h 1 year.
  • Storage grows with cardinality and retention. Limit labels; drop high-cardinality metrics if needed.
  • Document retention per environment. Align with compliance and capacity planning. Test queries on downsampled data.

Prerequisites

Steps

  1. Set retention

    Configure retention in Prometheus, Thanos, or vendor. Set raw and downsampled retention. Consider storage cost.

  2. Downsample if needed

    Create recording rules or jobs for 1m, 1h aggregates. Use for long-term dashboards and capacity.

  3. Document and review

    Document retention per env. Review storage growth; adjust retention or cardinality.

Summary

Set retention for raw and downsampled metrics; control cardinality; document and review storage.

Prerequisites

Steps

Step 1: Set retention

Configure retention in metrics backend; consider storage.

Step 2: Downsample if needed

Add recording rules or jobs for aggregates; use for long-term views.

Step 3: Document and review

Document retention; review storage and adjust.

Verification

  • Queries work for desired range; storage within budget.

Troubleshooting

Storage full — Reduce retention or cardinality. Missing old data — Check downsampling and retention rules.

Next steps

Continue to