Scalability and resilience

• Elasticsearch Scalability and Resilience:

  • Elasticsearch is distributed by nature, ensuring high availability and scalability.
  • Add servers (nodes) to a cluster to increase capacity; Elasticsearch distributes data and queries across nodes.
  • No application overhaul needed; Elasticsearch balances multi-node clusters automatically.
  • Under the hood, an index consists of one or more physical shards, each being a self-contained index.
  • Shards distribute documents across nodes for redundancy and query capacity.
  • Two types of shards: primaries (store documents) and replicas (copies of primaries).
  • Replicas serve read requests (searching or retrieving documents).

• Shard Considerations:

  • Shard size and number impact performance.
  • Aim for average shard size between a few GB and a few tens of GB.
  • Avoid excessive shards per node (proportional to heap space).
  • Test with your own data to find optimal configurations.

• Cross-Cluster Replication (CCR):

  • Synchronize indices from primary to secondary remote clusters.
  • Secondary clusters serve as hot backups or read-only followers.
  • Active-passive replication; primary handles write requests, secondary is read-only.
  • Use CCR for disaster recovery and geo-proximity read requests.