I need to write a post about my love for Elasticsearch and custom queries. On one project, our data pipeline wasn't performing as expected and it was logging each step to Elasticsearch and flexible queries that led me to realize that each step in the pipeline was working exactly as expected; it was that the amount of resources devoted to each step of the pipeline that was off (we were using Apache Storm and the parallelism devoted to certain areas was slightly off, creating a cascading effect).
I need to write a post about my love for Elasticsearch and custom queries. On one project, our data pipeline wasn't performing as expected and it was logging each step to Elasticsearch and flexible queries that led me to realize that each step in the pipeline was working exactly as expected; it was that the amount of resources devoted to each step of the pipeline that was off (we were using Apache Storm and the parallelism devoted to certain areas was slightly off, creating a cascading effect).
I would love to read that post!