How Bees runs team work without a central dispatcher

This post first appeared on the Bees blog.

Most systems that run jobs on many machines have a dispatcher. That is a central server that keeps a queue of jobs, picks a machine for each one and hands the job over. It works well when every machine is the same and any of them can do any job.

Bees computers are not the same. A Bees agent runs on your computer, next to your files, your sign-ins and the add-ons you connected. A report that reads a shared folder and your Gmail can only run on a computer where that folder is linked and the Gmail add-on is connected. A server far away cannot know that as well as the computer itself does.

So in Bees, each computer decides what it can run. Our server, Bees Cloud, does two small jobs. It keeps a lock so two computers do not run the same work, and it keeps a shared record so your team can see where the work is. It never tells a computer to start anything.

The parts

A few words first, because the rest of the post uses them.

• A process is a list of stages, like Draft, Review and Done.
• A work item is one piece of work moving through a process. One weekly report is one work item.
• An agent is an AI worker with its own instructions, model and tools. An agent does each stage.
• A schedule creates a new work item at a set time, like every Monday at 9:00.

Each computer runs its own copy of Temporal, an open-source workflow engine. Temporal writes down which stage each work item is at, so an app restart does not lose its place. Bees keeps that record in a file on your disk. There is no shared Temporal server.

What each computer decides

Everything about running the work happens on the computer.

Whether it can run the item at all. Before work starts, the computer checks itself. It needs an enabled agent for the work. The add-ons that agent uses must be connected on this computer. The folders the item reads must be linked to real folders here. If any check fails, the work waits and the computer checks again on its next pass.

Which agent does each stage. The computer picks in this order: the agents named on the work item (except for a review stage), then the agents set for that stage in the process, then the workspace's default agent, then the first enabled agent by name. A review stage never uses the agent that did the work it is reviewing. The computer saves each pick with its reason, so you can see later why an agent got the work.

What happens when an agent is busy. An agent can have a limit on how many runs it does at once. If it is at that limit, the stage waits and checks again every 15 seconds. After 10 minutes it shows as failed, so a person can see it and press Retry.

Moving between stages. When an agent finishes a stage, the item moves to the next one. When a reviewer asks for changes, the item goes back to the stage that did the work. The third time a reviewer asks for changes, the item stops and waits for a person. A stage that fails also waits for a person to press Retry.

Approvals. You can mark a stage to need a person's approval. On that stage the agent cannot send, post, delete or pay through an add-on until someone approves, and the stage cannot finish without it. The agent asks in the Bees app on the computer running the stage, and the run pauses until someone answers it there.

What Bees Cloud holds

The server keeps three things for this.

• An owner for each work item that runs. When a computer starts a work item, Bees Cloud records that computer and the person signed in on it as the owner. This lock does not run out and does not need renewing. It stays with that computer until the owner hands the run over. No other computer, yours or a teammate's, can take it in the meantime.
• A key for each scheduled time. When a schedule fires, the computer asks for a key made from the schedule and the time. The first to ask gets it. That key never runs out, so the same 9:00 can never run twice for the same person.
• The shared record. Every 15 seconds each computer syncs: it sends the stage and status of its work items and a summary of each run, and fetches what the rest of the team sent. The newest version wins, going by the clock on the computer that made the change.

Schedule keys are kept per person, so when two of your computers ask for the same 9:00, only one of them gets it. The owner of a work item is kept per team, so it holds against everyone on the team.

The server only answers yes or no. It has no queue, it never picks a computer and it never moves work anywhere. Computers call it. It never calls them. If you do not use Bees Cloud at all, there is nothing to ask, and your computer skips the lock and runs the work.

Handing a run over

A run only moves when its owner moves it. On the computer that owns the run, the owner presses Relinquish control. Bees stops the run and any work it handed to other agents. It writes a checkpoint file into the run's folder, and tells Bees Cloud the run is free and which stage each piece restarts at.

Then someone presses Continue on this machine on another computer. That can be another of your computers or a teammate's. That computer needs the run's folder, either through a shared folder linked on both computers or as a copy from the owner. It checks that the checkpoint and every file in it arrived unchanged, and runs the same checks as any other start. Only then does it claim the run and start the unfinished stage again, using the saved files. A review stage restarts from the stage that made the work being reviewed.

A real example: Maya's Monday report

Maya has a process called Weekly report with three stages: Draft, Review and Done. She scheduled it for every Monday at 9:00. She uses Bees on a laptop and a desktop, both signed in as her. Sam is on her team.

1. 9:00. Both of Maya's computers fire. The schedule has synced to both. Each one's Temporal fires it at 9:00, and each asks Bees Cloud for the key for this report at Monday 9:00.
2. The laptop asks first and gets the key. The desktop is told no and skips this Monday. Nothing else happens on the desktop.
3. The laptop creates the work item and claims it. Right away it sends the new item to Bees Cloud and becomes its owner. Then it starts Draft. It does not need to renew anything.
4. The desktop sees the item arrive. On its next sync, within 15 seconds, it shows the report as running on Maya's laptop. It never asks to run it.
5. Sam sees the report in his Bees. His app shows the report at Draft, but his computer never runs it. A team schedule only fires on the computers of the person who owns it, and his app never asks for a run that another computer owns.
6. 9:04. Maya closes the laptop lid. The laptop sleeps in the middle of Draft. Bees Cloud does nothing about it and sends nothing to anyone. The report still belongs to the laptop, so the desktop does not start it.
7. 9:20. Maya opens the laptop. The laptop still owns the report, and Draft carries on there. If the sleep cut the agent off, Draft shows as stopped until Maya presses Retry.
8. 9:30. Maya has to leave and wants the desktop to finish. On the laptop she presses Relinquish control. Bees stops Draft, saves the checkpoint in the run's folder and tells Bees Cloud the report is free.
9. Maya presses Continue on this machine on the desktop. The run's folder is a shared folder linked on both computers, so the checkpoint and files are already there. The desktop checks them, claims the report and starts Draft again from the top. It picks the agent the same way the laptop did.
10. The desktop finishes the report. Draft, then Review, then Done. Each move syncs, so Sam watches the report move along. If a stage needs Maya's approval, she answers it in Bees on the desktop, because that is where the stage is running.

At no point did the server choose the desktop. Maya chose it, and the desktop checked itself before it took the run.

What this costs

• Work waits for the computer that owns it. Nothing on our side can run your work. A run waits while its computer is asleep or offline, even if another of your computers is free. Only the owning computer can hand it over, so a laptop that is lost or broken keeps its runs. The other computer also has to pass the same checks. If the report reads a folder that is only linked on the laptop, the desktop cannot continue it.
• A missed time is skipped. If none of your computers is awake with Bees running at 9:00, or the first one wakes more than a minute late, that Monday's report does not run. It does not catch up later. A laptop opened at noon will not fire a pile of old runs, but you also lose that week's run.
• A handover repeats the stage. The new computer starts the unfinished stage from the top, using the files saved in the run's folder. Files never go through our server, so that folder has to reach the other computer through a linked shared folder or a copy. A handover takes at most 1,000 files, 20 MB each and 250 MB in all. If the stage had already sent an email, the new computer may send it again.
• Nothing moves on its own. An earlier version handed a run to another computer when the first one went quiet for a minute. That meant two computers could run the same stage at once, after a Wi-Fi drop or a laptop waking up. Now a person has to hand the run over, so a stage never runs in two places, but a sleeping laptop holds its work until it wakes.
• The shared record trusts each computer's clock. When two computers change the same item, the change with the later time wins, and that time comes from the computer's own clock. A computer with a wrong clock can win when it should not.
• Team schedules need our server at the scheduled time. A computer that cannot reach Bees Cloud at 9:00 cannot get the key. It tries five times and then gives up on that run.
• The server still sees some things. Each run owner records a random id for the computer, the computer's name, the person and the work item it covers. A handover also records which stage each piece restarts at. Owners and schedule keys are never deleted, so the server keeps one record for every run and every scheduled time each person has claimed. The shared record holds the names, titles, stages, status and run summaries described in our last post.

A question for you

Making a person hand a run over, and restarting the half-done stage from the top, is the simplest thing that never runs a stage twice at once. It is also the part we are least sure about. If you have run work on machines that sleep, like laptops, what did you do with a step that was half done when the machine went away: move it automatically and accept the duplicates, save checkpoints inside the step, or keep work on one machine? Tell us what held up on Discord.

If you want to try Bees on your own machines, download it here.