Hi Molly. Thanks for this article, very useful. Can you say what number of ES nodes do you use, now and historically? How much did this number helped your performance compared to optimizations you wrote about?
We had 21 nodes before we made these optimizations, amongst others, at that time we had 750 million documents, processing a 10-20 million a day
We now have 22 nodes in our cluster which holds over 5 billion documents and processes over 500 million of them a day.
Making optimizations to our data structure and the way we handled the data allowed us to scale without having to increase the size of our cluster. The extra node was added very recently to added additional space for our growing set of documents. We haven't seen really any big performance boost from having it.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Hi Molly. Thanks for this article, very useful. Can you say what number of ES nodes do you use, now and historically? How much did this number helped your performance compared to optimizations you wrote about?
We had 21 nodes before we made these optimizations, amongst others, at that time we had 750 million documents, processing a 10-20 million a day
We now have 22 nodes in our cluster which holds over 5 billion documents and processes over 500 million of them a day.
Making optimizations to our data structure and the way we handled the data allowed us to scale without having to increase the size of our cluster. The extra node was added very recently to added additional space for our growing set of documents. We haven't seen really any big performance boost from having it.