This document describes how sharding internally works in MongoDB. It will be useful to anyone who’s into trying to understand the internal parts of the MongoDB sharding.
This same information can be found in the MongoDB official documentation arranged in a different way. I just crafted it in a way that helped me into trying to understand how MongoDB sharding internally works.
How is data sharded
MongoDB shards data at the collection level, distributing the collection data across the shards in the cluster.
MongoDB uses the shard key to distribute the collection's documents across shards. The shard key consists of a field or multiple fields in the documents.
MongoDB partitions sharded data into chunks. Each chunk has an inclusive lower and exclusive upper range based on the shard key.
Sharding Strategies
MongoDB supports two sharding strategies:
- Hashed sharding: it involves computing a hash of the shard key field's value. Each chunk is then assigned a range based on the hashed shard key values.
- Range sharding: it involves dividing data into ranges based on the shard key values. Each chunk is then assigned a range based on the shard key values.
How is data evenly distributed
The MongoDB balancer is a background process that monitors the amount of data on each shard for each sharded collection. When the amount of data for a sharded collection on a given shard reaches specific migration thresholds, the balancer attempts to automatically migrate data between shards and reach an even amount of data per shard.
The server schedules an auto-split if:
- It detected that the chunk exceeds a threshold based on the maximum chunk size.
- There is not already a split in progress for the chunk.
- Every time an update or insert gets routed to a chunk, the server tracks the bytes written to the chunk in memory through the collection's ChunkManager.
Note that because the balancer keeps distribution status per collection it optimizes for an even distribution per collection rather than for the entire cluster.
How are requests routed to the correct shard
The Config servers store the metadata for a sharded cluster. The metadata reflects state and organization for all data and components within the sharded cluster. The metadata includes the list of chunks on every shard and the ranges that define the chunks.
The config database contains the collections that contain the sharded cluster metadata. MongoDB writes data to the config database when the metadata changes, such as after a chunk migration or a chunk split.
- The config.chunks collection stores a document for each chunk in the cluster.
- The config.collections collection stores a document for each sharded collection in the cluster.
- The config.databases collection stores a document for each database in the cluster. Each database in a sharded cluster has a primary shard that holds all the un-sharded collections for that database. Each database has its own primary shard.
- The config.shards collection represents each shard in the cluster in a separate document,
The mongos instances cache this data and use it to route read and write operations to the correct shards. mongos updates the cache when there are metadata changes for the cluster, such as adding a shard. Shards also read chunk metadata from the config servers.
The cache may or may not be up-to-date with its corresponding source.
The routing table cache is “lazy.” It does not refresh from its source unless necessary.
Operations that change a collection’s routing information (for example, a moveChunk command that updates a chunk’s location) will mark the local node’s routing table cache as “stale” for affected shards. Subsequent attempts to access routing information for affected shards will block on a routing table cache refresh.
Top comments (0)