Import CSV data into Elasticsearch

For a one-off import, Kibana's file upload workflow is convenient. For repeatable ingestion, use Logstash or a program built on the bulk API. Decide field types and document identity before importing; a successful upload does not guarantee that dates and numbers were mapped as intended.

Upload a file in Kibana

Find Upload a file through the navigation or global search, select the CSV, review detected delimiters and field types, and inspect the proposed mapping before importing. Location and limits vary by release and deployment. Current Elastic documentation describes a default limit of 500 MB for CSV and similar text formats; older installations may expose a different limit.

This workflow is intended for exploratory or occasional imports, rather than a continuously running production ingestion pipeline.

Parse CSV with Logstash

For a headered file with one record per physical line:

input {
  file {
    path => "/data/sales.csv"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/sales.sincedb"
  }
}
filter {
  csv {
    separator => ","
    columns => ["date", "product", "amount", "region"]
    skip_header => true
  }
  mutate {
    convert => { "amount" => "float" }
  }
  date {
    match => ["date", "yyyy-MM-dd"]
    timezone => "UTC"
  }
}
output {
  elasticsearch {
    hosts => ["https://es.example.com:9200"]
    index => "sales-%{+yyyy.MM}"
  }
}

Run this pipeline with one worker because the CSV filter's header-skipping behavior requires it. Use the source data's actual date time zone. Add the deployment's authentication and TLS settings, and route parse failures for inspection before relying on the imported data.

The persistent sincedb_path remembers progress. start_position affects previously unseen files; it does not override stored offsets. Using /dev/null for state can replay files after restart and create duplicates. For a deliberate replay, choose a separate destination or stable document IDs. A line-oriented file input is also unsuitable for CSV records containing quoted embedded newlines without additional handling.

Import with Python

Python's CSV parser can handle quoted records, including embedded newlines:

import csv
from elasticsearch import Elasticsearch, helpers

es = Elasticsearch('http://localhost:9200')  # Local test cluster only.

def actions():
    with open('sales.csv', newline='', encoding='utf-8-sig') as stream:
        for row in csv.DictReader(stream):
            row['amount'] = float(row['amount'])
            yield {'_index': 'sales', '_source': row}

helpers.bulk(es, actions())

Create the destination mapping first, including a date mapping for date. Add authentication for a secured cluster. utf-8-sig accepts UTF-8 with or without a BOM; other encodings need explicit conversion. CSV readers return strings until you convert them.

Make large imports recoverable

Tune bulk sizes against measured response time and memory, handle failed items, and define retry/deduplication behavior. Temporarily changing refresh behavior can help a controlled bulk load, but restore it afterward and understand that search visibility is delayed. Third-party loaders are another option only after checking compatibility and maintenance.

References: Kibana file upload and Logstash CSV filter.