Skip to content

Elasticsearch ILM policies

Learn how to write an Elasticsearch ILM (Index Lifecycle Management) policy that moves log indices to cheaper storage as they age and deletes them when they expire. You will build a working logs policy step by step, from the rollover trigger to the delete phase.

This guide shows operators of Elasticsearch clusters that store logs or metrics how to set up Index Lifecycle Management (ILM). Those indices (Elasticsearch’s name for a collection of documents, much like a database table) grow every day until disks fill and someone has to delete old data by hand. ILM is the built-in fix: you write a policy once, and Elasticsearch applies it to each index as it ages. The example policy starts a fresh index when the current one gets too large or turns 7 days old (this is called rollover), then moves older indices to cheaper warm and cold storage and deletes them after 180 days. By the end you will also have the index template that attaches the policy to every new log index.

How to verify

After creating a policy and binding it via an index template, confirm:

curl -sk -u elastic:"$PW" 'https://es:9200/_ilm/policy/logs-policy?pretty'
curl -sk -u elastic:"$PW" 'https://es:9200/_cat/indices/logs-*?v&s=index'
curl -sk -u elastic:"$PW" 'https://es:9200/logs-000001/_ilm/explain?pretty'
curl -sk -u elastic:"$PW" 'https://es:9200/_ilm/status'

Healthy: status RUNNING, indices show their current phase, no step_info.type: ERROR in _ilm/explain.

What’s happening

ILM works on time-series indices that follow a rollover pattern: a write alias points to the current “hot” index, and when a trigger fires, a new index is created and the alias swings over. ILM doesn’t manage one giant index — it manages a chain of them.

Five phases exist:

  • Hot — actively written, all replicas, refresh enabled. Lives on fast disk (SSD nodes with the data_hot role).
  • Warm — read-only, replicas reduced, forcemerged to fewer segments. Lives on data_warm nodes.
  • Cold — read-only, often shrunk to single shard, possibly searchable_snapshot. Lives on data_cold.
  • Frozen — fully on object storage via searchable snapshots. Lives on data_frozen.
  • Delete — index is gone.

Each phase has a min_age (since rollover, not since creation) and an actions block. Actions are atomic operations: rollover, forcemerge, shrink, allocate, searchable_snapshot, freeze, delete.

The policy is bound to indices via an index template’s index.lifecycle.name setting. Filebeat/Logstash write to the alias; ILM creates the next index when the trigger fires.

The procedure

1. Create the ILM policy for logs:

curl -sk -u elastic:"$PW" -X PUT 'https://es:9200/_ilm/policy/logs-policy' \
  -H 'Content-Type: application/json' \
  -d '{
    "policy": {
      "phases": {
        "hot": {
          "actions": {
            "rollover": {
              "max_primary_shard_size": "30gb",
              "max_age": "7d"
            },
            "set_priority": {"priority": 100}
          }
        },
        "warm": {
          "min_age": "7d",
          "actions": {
            "forcemerge": {"max_num_segments": 1},
            "allocate": {"number_of_replicas": 1, "include": {"data_tier": "data_warm"}},
            "set_priority": {"priority": 50}
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {
            "allocate": {"number_of_replicas": 0, "include": {"data_tier": "data_cold"}},
            "set_priority": {"priority": 0}
          }
        },
        "delete": {
          "min_age": "180d",
          "actions": {"delete": {}}
        }
      }
    }
  }'

Rollover triggers when the primary shard reaches 30 GB OR the index reaches 7 days, whichever first. Age-based fallback prevents stale tiny indices from sitting in hot tier forever.

2. Create the index template that binds the policy:

curl -sk -u elastic:"$PW" -X PUT 'https://es:9200/_index_template/logs-template' \
  -H 'Content-Type: application/json' \
  -d '{
    "index_patterns": ["logs-*"],
    "template": {
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1,
        "refresh_interval": "30s",
        "index.lifecycle.name": "logs-policy",
        "index.lifecycle.rollover_alias": "logs"
      }
    },
    "priority": 200
  }'

number_of_shards: 1 is correct for most log workloads — over-sharding is the most common ILM-adjacent mistake.

3. Bootstrap the first index and alias:

curl -sk -u elastic:"$PW" -X PUT 'https://es:9200/logs-000001' \
  -H 'Content-Type: application/json' \
  -d '{
    "aliases": {
      "logs": {"is_write_index": true}
    }
  }'

This is the one-time bootstrap. From here on, ILM creates logs-000002, logs-000003, etc., automatically.

4. Verify rollover triggers manually (without waiting):

curl -sk -u elastic:"$PW" -X POST 'https://es:9200/logs/_rollover' \
  -H 'Content-Type: application/json' \
  -d '{"conditions": {"max_age": "1m"}}'

This forces a rollover for testing. Check that logs-000002 appears.

5. Inspect the lifecycle status of any index:

curl -sk -u elastic:"$PW" 'https://es:9200/logs-000001/_ilm/explain?pretty'

Look for phase, action, step, and phase_execution.policy. If step is stuck at check-rollover-ready while the trigger conditions are met, the alias may be misconfigured.

6. Resolve a stuck index:

curl -sk -u elastic:"$PW" -X POST 'https://es:9200/logs-000001/_ilm/retry'

If a phase action failed (e.g., insufficient warm-tier nodes), fix the underlying cause, then retry. ILM will resume from the failed step.

Common pitfalls

  • Missing rollover alias is the #1 issue. ILM rolls over via the alias — without it, indices just sit. Always bootstrap a write alias.
  • max_age only, no max_primary_shard_size — leads to giant indices when traffic spikes. Always combine.
  • Forcemerge in hot phase — never. It’s expensive and produces unbalanced segments. Only forcemerge in warm.
  • Cold phase without data_cold nodes — the policy will stall waiting for allocation. Either remove the tier requirement or add the nodes.
  • Time math. min_age: 7d is from rollover time, not index creation. A hot index that doesn’t roll because of low traffic stays hot.
  • Replica count of 0 in warm/cold — saves disk but loses redundancy. If your snapshots are tight, accept the trade-off; if not, keep one replica until cold.

In the engagements we run, ILM is the difference between a search cluster that runs itself and one that the team babysits. The hard work is in the design: shard sizing, tier sizing, snapshot cadence, and retention have to match the business need. Once tuned, the cluster scales linearly with log volume and there’s no manual cleanup. See /en/services/managed-operations/ for how we operate ILM-managed search clusters at scale.