Loading CSVs to elasticsearch like a pro (no Logstash needed)

#elasticsearch #logstash #devops #database

As an Elasticsearch user, I subscribed to elasticsearch groups and channels in many places (Twitter, Reddit, Facebook, StackOverflow) and I keep seeing people struggling to start with elasticsearch, especially loading bunch of CSV files into it.

Hey does anyone know how can i upload multiple CSV files to Kibana?

The first mistake as I see it is to use Logstash in order to load CSV files to elasticsearch.
I see Logstash as a streaming service to elasticsearch rather than a tool to load data to elasticsearch from a static source (like CSV).

Using Elasticsearch Loader to load CSVs to Elasticsearch

I wrote a wrapper on top of Elasticsearch's bulk API in order to reduce the barrier in loading files into Elasticsearch.
The usage simple as running a command from your terminal.

Install

pip install elasticsearch-loader

Plugins

There are several plugins like Parquet, Redis, and S3 as input sources, read more about them in the readme

Usage and examples

Load CSVs into elasticsearch running on localhost
elasticsearch_loader --index incidents csv file1.csv file2.csv

Load CSV from URL and specify id field
elasticsearch_loader --index data --id-field state csv https://raw.githubusercontent.com/jakevdp/data-USstates/master/state-abbrevs.csv

Read CSV data from stdin
generate_data | elasticsearch_loader --index data csv -

And much more, for more examples and documantaion please follow elasticsearch-loader's README file