Three houses, a lane full of trick-or-treaters, and a Halloween night with no one in charge.
An Amazon OpenSearch Service domain does two kinds of work. It answers the requests your applications send, and it runs itself: choosing a leader, deciding which node holds which shard, and publishing every change to the cluster state. When a busy domain slows down, the usual fix is more or bigger data nodes, but the slowdown often comes from the second kind of work landing on the same nodes as the first. I first wrote a version of this story for my colleagues, and Halloween is the right night to bring it back, because it makes the answer easy to see: three small dedicated cluster manager nodes that give the domain its own command.
Halloween night on Hollow Lane
Every Halloween, the village of Hollowmere pools all of its candy into one shared stash and hands it out from three houses on Hollow Lane. The stash is far too big for one porch, so it sits in bowls spread across Kael's house, Mira's house, and Dorn's house. Every bowl has a twin on a different porch, so a knocked-over bowl never means lost candy. Any host can answer any knock. When a visitor wants something from a bowl on another porch, the host at the door shouts down the lane, the other hosts check their bowls, and the host hands over the whole haul.
The visitors were the real challenge of the night, and their costumes told you what each one wanted. Ghosts drifted up with sacks of candy to add to the stash, and every piece had to land in the right bowl and in that bowl's twin. Pirates came hunting treasure: one particular kind of candy, wherever on the lane it happened to sit. Vampires, who in the old stories can never resist counting, wanted only tallies, such as how many chocolate bars were left, broken down by brand. They arrived by the hundreds, and the rule of the night was old and strict. Answer fast with a treat, or get a trick.
The hosts had the candy, the bowls, and plenty of energy. What the lane did not have was anyone in charge of the night.
For the first hour the hosts kept pace. Then the village council came out with lanterns and orders, and with no one else to take them, the council went straight to the hosts. Some orders changed how the night worked. “Start a new bowl just for gummy bats!” called the mayor. “Every candy needs a label saying whether it has nuts!” said the baker. “Two pieces per pirate from now on!” cried the schoolteacher. Each order meant all three hosts had to stop, take it in, and change how they worked, in the middle of answering the door.
Other orders were bigger, and they cost more. “Who is in charge out here?” demanded the blacksmith, so the hosts had to stop and vote on a captain. “Which house keeps the caramel apples?” asked the miller, and they had to agree, then shuffle bowls when the next councilor disagreed. Then Dorn's porch light flickered out, and the other two had to work out which twin bowls now counted as the real ones and where to set out fresh twins, while the doorbells kept ringing.
The visitors did not wait. Ghosts stood on the steps holding sacks no one had time to sort. A pirate waited while Kael shouted down the lane about bowl assignments instead of caramel apples. A vampire waiting on his tally gave up and started counting the gravel. Every visitor who waited too long played a trick, and by midnight Hollow Lane was draped in toilet paper. The hosts had not failed at handing out candy. The night had failed because no one was running it.
Translating Hollow Lane
In Amazon OpenSearch Service, Hollow Lane is your domain and the three houses are its data nodes. The candy is your data, the bowls are shards, and the twin bowls are replicas, placed on different nodes so that losing one node never loses data. The visitors are the requests your applications send: ghosts are indexing requests that add documents, pirates are search queries, and vampires are aggregations. Any data node can take a request and act as its coordinator, sending it to whichever nodes hold the relevant shards and gathering their answers into one response. A treat is a fast response. A trick is a slow or failed one.
The council's orders are the domain's management work. The tactical orders are changes to the cluster state: a new bowl for gummy bats is creating an index, the nut labels are a mapping update, and two pieces per pirate is a settings change. Each change produces a new version of the cluster state that every node has to receive and apply. The strategic orders are cluster management itself: electing the cluster manager, deciding which node holds which shard, noticing when a node goes dark, promoting replicas to take its place, and computing every new version of the cluster state. Those are jobs for the domain as a whole, not for any one node. Without dedicated managers, the data nodes are the only candidates, so they do this work between requests.
On a quiet lane that overhead barely registers. The trouble grows with the domain. As shard counts climb, mappings get larger, and indexes come and go more often, the council's orders multiply, and data nodes spend more time on management and less on requests. Latency rises, indexing throughput drops, and the tricks pile up. Under heavy load it can turn into a loop: data nodes get too busy to apply cluster state updates promptly, the elected manager retries, the retries add load, and the added load slows the data nodes further.
The witches in the bell tower
The next Halloween, Hollow Lane had someone in charge. Three witches climbed the bell tower at the end of the lane. They handed out no candy and answered no doors. From the tower they could see every porch, and they kept the master list of which bowl sat where. Every councilor with an order now climbed the tower stairs instead of knocking on a host's door.
Among themselves, the witches ran the night. One led, and the other two stood ready to take over if she fell. They decided which porch kept which bowl and moved bowls only when the lane changed. They watched every porch light, and when one went out they decided which twins took over and where the new twins went. They turned the council's requests into one clear instruction at a time and called it down the lane along with a fresh copy of the master list. The hosts heard each call, adjusted, and went back to the door, and because every house had the latest list, a host always knew which porch to shout to when a pirate asked for caramel apples. Nobody on the lane stopped to vote, and the toilet paper stayed at the store.
Dedicated cluster managers in practice
Dedicated cluster manager nodes are the witches: the command role for the whole domain, held by nodes that do nothing else. They hold no data, serve no queries, and index no documents. Their job is coordination: tracking every node, index, and shard, maintaining routing information, computing a new cluster state when indexes or nodes come and go, publishing that state to every node, and sending heartbeats to confirm the data nodes are healthy. The published cluster state is the master list. Every data node gets a copy, which is how a coordinating node knows where to send each request. Data nodes still apply each update, the way the hosts adjusted to each call from the tower, but they no longer compute it or vote on who leads. You will find the setting under its older name in the console, where these are still called dedicated master nodes.
For production domains, AWS recommends three dedicated managers, and the reason is quorum. Only one manager is active at a time, while the other two wait as standbys. If the active one fails, the remaining two still form a majority and elect a replacement without pulling in the data nodes. Two managers cannot do that, because losing one leaves no majority, and without quorum the domain fails both reads and writes. Four buys nothing over three and can lose quorum when an Availability Zone goes down, which is why AWS never recommends an even number. When you deploy with Multi-AZ with Standby, the service provisions the three managers across three zones for you.
Because managers coordinate rather than store, you size them by how much they have to track, not by how much data you hold. The AWS guidance scales manager memory with the number of nodes and shards in the domain. On OpenSearch 2.17 and later, an 8 GB manager handles up to 30 nodes and 15,000 shards, and a 32 GB manager handles up to 120 nodes and 60,000 shards. Domains with many shards or large field mappings may need the next size up. Either way, the managers cost a small fraction of the data tier, which is a modest price for the stability they buy back.
Bringing it together
An Amazon OpenSearch Service domain has two kinds of work: answering the requests your applications send, and running the domain itself. Without dedicated cluster managers, nobody is assigned the second job, so the data nodes answer the door, carry out every change to the cluster state, and argue out who leads and where each shard lives, all at once. Dedicated cluster manager nodes give the domain its command. They take the management work away from the data nodes and turn every change into a published update the data nodes only have to apply. Three of them give you quorum, so a failed manager is replaced without the data nodes stopping to vote, and they run on instances sized to the domain's node and shard count rather than its data volume. If you run anything close to a production workload, put your witches in the tower before the first knock.
Top comments (1)
The three witches with one leading and two ready to step in makes the quorum point stick.