Delete old Elasticsearch indices with a retention policy

For recurring retention, use the lifecycle mechanism supported by your deployment. For a one-off cleanup, review the exact index names and delete only the intended indices. Index deletion removes all their documents; recovery requires a usable snapshot or another retained copy of the data.

Review a manual deletion

In Kibana Dev Tools:

GET _cat/indices?v&s=creation.date&h=index,creation.date.string,docs.count,store.size

An index's name or creation date does not prove every document inside it is older than your retention cutoff. Check its lifecycle, aliases, contents, and backup coverage. Then delete an explicitly reviewed name:

DELETE logs-2024.01.01

Do not delete an active write index. A data stream's current write index has additional protections and must be handled through the appropriate rollover/lifecycle workflow.

Configure ILM with a rollover alias

This example is for a deployment supporting ILM and ordinary indices with an alias. It is not an Elasticsearch Serverless configuration; Serverless uses data lifecycle management rather than ILM.

Create the policy:

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "1d", "max_primary_shard_size": "30gb" }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Create a matching template, then bootstrap the write alias:

PUT _index_template/logs_template
{
  "index_patterns": ["logs-app-*"],
  "priority": 200,
  "template": {
    "settings": {
      "index.lifecycle.name": "logs_policy",
      "index.lifecycle.rollover_alias": "logs-write"
    }
  }
}

PUT logs-app-000001
{
  "aliases": {
    "logs-write": { "is_write_index": true }
  }
}

Add your mappings and appropriate shard settings to the template, and ensure its pattern and priority do not collide with existing templates. Send subsequent writes to logs-write. Merely attaching a rollover policy without the alias and bootstrap index is insufficient.

After rollover, phase ages are measured from rollover time. Thus the delete phase becomes eligible 30 days after rollover, not exactly 30 days after each document's timestamp. Empty indices do not normally roll over on age alone, and lifecycle execution is periodic. Monitor progress with GET logs-app-*/_ilm/explain.

Maintain an existing Curator workflow

For a compatible Curator installation, an action file can select by prefix and creation age:

actions:
  1:
    action: delete_indices
    description: Delete reviewed logs-archive indices older than 30 days
    options:
      ignore_empty_list: true
      disable_action: false
    filters:
      - filtertype: pattern
        kind: prefix
        value: logs-archive-
      - filtertype: age
        source: creation_date
        direction: older
        unit: days
        unit_count: 30

Test selection with curator --config curator.yml --dry-run action.yml, inspect the selected names, and only then schedule the real command. Keep the prefix restricted to inactive archives.

Deleting an entire index removes its shard files; it does not wait for document-deletion segment merges. Filesystem space reporting can lag cleanup or open file handles. Force merge is not a way to accelerate cleanup of an index that no longer exists.

References: ILM without data streams, rollover age rules, and Curator delete indices.