Count unique field values in Kibana and Elasticsearch

Use Elasticsearch's cardinality aggregation, or Kibana's Unique count operation, to estimate the number of distinct field values. The result uses HyperLogLog++ and is approximate; there is no universal fixed error percentage for every dataset.

Build the metric in Kibana

Create a metric visualization, select Unique count or the equivalent distinct-count operation, and choose an aggregatable field. For an identifier mapped as keyword, use that field directly. If it is a text field with a keyword subfield, use the subfield, such as user_id.keyword.

Not every string has a .keyword subfield. Inspect the mapping instead of appending the suffix automatically. Numeric identifier fields can also be aggregated. Avoid enabling expensive text fielddata merely to count whole identifiers.

Query Elasticsearch directly

GET logs-*/_search
{
  "size": 0,
  "aggs": {
    "unique_users": {
      "cardinality": {
        "field": "user_id.keyword",
        "precision_threshold": 40000
      }
    }
  }
}

Add the same time and other filters used by the dashboard if you want comparable results. Otherwise this example covers all documents matched by logs-*.

precision_threshold trades memory for accuracy. The documented default is 3,000 and the maximum effective value is 40,000. A higher value is not an exact-count switch, and the quality of an estimate depends on the data distribution. Missing values are normally ignored unless you configure a substitute.

Group by day

Place the cardinality aggregation inside a date_histogram to calculate daily distinct counts. A user seen on two days belongs to both daily counts, so summing daily values does not give the distinct count for the whole period. Decide the time zone and calendar boundary explicitly.

When exactness matters

For an exact result, one option is to enumerate distinct values with a paginated composite aggregation and count the returned buckets. This is a different workload with higher cost; use a consistent dataset or a supported consistent-view strategy if writes continue during pagination. A single terms aggregation with a large size is not a generally reliable substitute for enumerating every value.

See Elastic's cardinality documentation and composite aggregation.