DEV Community: Temporal

Why I needed Durable Execution to Read a Toy Manual

Shy Ruparel — Fri, 10 Apr 2026 15:53:01 +0000

Watch me take a Japanese toy manual and turn its translation into a bulletproof, AI-powered ETL pipeline. I’ll show you how I use Temporal Workflows to guarantee an AI pipeline never loses progress, surviving network failures, API crashes, and more.

What You'll Learn

Why Spider-man is the reason that Power Rangers has a giant robot.
Guaranteed Completion with Temporal: I’ll show you how to ensure your code keeps running even if servers crash or APIs fail.
Parallel OCR & Translation: Learn how I used a "fan-out" pattern with Google Document AI to process a 20-page manual in 50 seconds instead of 10 minutes.
Resilient AI Cleanup: See how I use Pydantic and Temporal together to handle non-deterministic LLM outputs from Gemini and automatically retry failed validations.

Ready to build it yourself? 👉 Check out the code here!

Decoupling Temporal Services with Nexus and the Java SDK

Nikolay Advolodkin — Thu, 02 Apr 2026 13:50:51 +0000

Your Temporal services share a blast radius. A bug in Compliance at 3 AM crashes Payments, too, because they share the same Worker. The obvious fix is separate services with HTTP calls between them - but then you're managing HTTP clients, routing, error mapping, and callback infrastructure yourself.

We published a hands-on tutorial on learn.temporal.io where you take a monolithic banking payment system and split it into two independently deployable services connected through Temporal Nexus.

You'll learn:

Nexus Endpoints, Services, and Operations from scratch
Two handler patterns for different use cases
How to swap an Activity call for a durable cross-namespace Nexus call

The caller-side change is minimal - the method call stays the same:

// BEFORE (monolith - direct activity call):
ComplianceResult compliance = complianceActivity.checkCompliance(compReq);

// AFTER (Nexus - durable cross-team call):
ComplianceResult compliance = complianceService.checkCompliance(compReq);

Same method name. Same input. Same output. Behind that swap: a shared service contract, a Nexus handler, an endpoint registration, and a Worker configuration change.

Here's what the Nexus handler looks like - it backs the operation with a long-running workflow so retries reuse the existing workflow instead of creating duplicates:

@OperationImpl
public OperationHandler<ComplianceRequest, ComplianceResult> checkCompliance() {
    return WorkflowRunOperation.fromWorkflowHandle((ctx, details, input) -> {
        WorkflowClient client = Nexus.getOperationContext().getWorkflowClient();
        ComplianceWorkflow wf = client.newWorkflowStub(
            ComplianceWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue("compliance-risk")
                .setWorkflowId("compliance-" + input.getTransactionId())
                .build());
        return WorkflowHandle.fromWorkflowMethod(wf::run, input);
    });
}

The tutorial includes a durability checkpoint: you kill the Compliance Worker mid-transaction, restart it, and watch the payment resume exactly where it left off. No retry logic, no data loss across the namespace boundary. Java SDK, runs entirely on Temporal's dev server.

Try it

2025 — Part 2

Sergey Bykov — Tue, 18 Nov 2025 16:55:03 +0000

(Part 1)

Company

At the time of my last update, the company had 116 people. Now we are over 300. The Go-to-Market organization is now larger than Engineering. Some studies claim that our ancestors couldn’t handle tribes of over about 150 people. We are definitely past the point when one could know every employee. The loss of intimacy is offset by the feeling that we now have resources — a growing number of teams focusing on different areas while collaborating on cross-group efforts.

With such growth, we are doubling down on our efforts to foster and reemphasize consistency in our hiring practices, decision-making, behavioral patterns, and rules of engagement, otherwise referred to as values and culture. In my previous life within a huge corporation, those things generally made sense to me, but they also felt somewhat artificial and performative. Within the context of a small company with a relatively flat structure, it feels very different — much closer to home. This makes me genuinely attentive to such aspects and eager to contribute where I can. Just recently, we rolled out our updated values.

My impression is that at least half of the VC money these days goes to companies with corporate domains ending in “.ai,” and aside from that, funding isn’t easy. We raised our C round early this year with a very good, some say almost exceptional, multiple. This tells us that the investors have a strong conviction about our product, business model, and growth. I’m no VC, but I see how they are impressed with the quality of the use cases and the caliber of customers that come to our cloud. I hope they know better than I do how to assess and evaluate such factors. Since the C round, we’ve also had a secondary round that pushed the company’s valuation significantly higher.

Keeping the hiring bar high continues to be a top priority. With the turmoil in the job market and Temporal becoming a better-known brand, we now have access to a larger pool of high-quality engineering talent. The interview process is still more art than science, and scaling and improving this art as the company grows is a challenge by itself. Hiring at the junior levels has its own difficulties. Recently, we had to close an open SDE 1 position after only a few hours because, during that time, we received more than 3,000 applications. We found that the old recipe still works well — filling junior positions via internships.

We are still fully remote, with WeWork as an option for folks who want to come into the office. We are geo-distributed but not very balanced. Most of Engineering is on the U.S. West Coast, with roughly a tie between the Seattle and Bay areas. Smaller pockets are in Colorado, North Carolina, and the cities of New York, Chicago, Toronto, and Vancouver. The GTM team has its own distribution. My impression is they are more heavily tilted toward the East Coast.

We settled on an annual all-company offsite (we started with twice a year). We complemented it with smaller team offsites and are now aggregating them into an annual R&D offsite, side by side with GTM’s sales kickoff event. We’ll see how this goes. There doesn’t appear to be a simple solution for doing it right, and each company needs to find its own rhythm. From time to time, we leverage the West Coast’s locality for in-person meetings to discuss some critical decisions or designs. In such cases, we consciously violate the remote-first setup for the sake of high-throughput discussions and faster decision-making — at the unfair expense of colleagues who can’t attend in person and have to connect via Zoom.

Replay

It was a bold move in August 2022 to start our own annual conference. The inaugural edition was in Seattle. The 2023 and 2024 editions were in Bellevue, WA, growing bigger each year. In 2025, we held the event in London to reach audiences unlikely to travel to the U.S. Attending Replay is a very special experience. Seeing so many engineers and engineering leaders talking non-stop about your product and presenting on stage what they’ve built with it is a special kind of pleasure. I presented at all Replays but the very first one. In 2023, my talk was on the second day and I talked with folks so much before then that my voice let me down close to the end of my presentation. I guess that’s why no recording of it was published. But I gave slightly different versions of the same talk at J on the Beach and QCon SF that year.

Replay 2026 will be in San Francisco — at Moscone, no less. It should be epic. I’ll need to rewatch the Silicon Valley documentary before going there.

Operations

We operate a multi-million-dollar business based on a single product — Temporal Cloud. Our customers trust us with their hot-path business processes — often their most critical ones. This is an interesting phenomenon. They choose Durable Execution of Temporal to make their applications resilient to various failures. Naturally, they first and foremost care about the reliability of their most critical services. Some choose to self-host Temporal Server with all its dependencies. Many don’t view it as their core competency — operating such complex production machinery — and they come to our cloud service with their most precious workloads. It is amazing and sobering at the same time when big Internet household names bring us their “crown jewels” to run — even those who have a policy of not taking a dependency on SaaS vendors in the critical path. It was eye-opening to hear, on a couple of occasions, a customer say, “We only have two external dependencies — AWS and Temporal Cloud.”

Customer expectations are very high. Sometimes it feels like they set them higher for us than for the hyperscalers. We now have about eight engineering on-call rotations (teams), covering different areas of the system, plus one for on-call managers who coordinate across teams, and another for the Developer Success team that communicates with customers. This may seem large for our company size, but that’s the nature of the service we run.

We use incident.io for managing incidents. It integrates nicely with Slack, creates a per-incident channel, and automatically adds the current on-call engineers to it, among other things. We saw great promise in the early days of their product. They haven’t disappointed and are growing fast. Like most folks, we use statuspage.io for public incidents and pagerduty.com for on-call paging. Incident.io also integrates with Jira to automatically turn incident follow-ups into tickets, helping us continuously improve the system.

Replication

Temporal inherited the application-level replication stack from Cadence. Over the years, we dramatically improved it and added Control Plane functionality to manage it. Initially, we used replication to transparently migrate customer namespaces from Cell to Cell. After we got it working at the level we were happy with, we exposed it to customers as high-availability options — multi-region, cross-cloud, and single-region replication.

At first, few customers immediately understood why they would want to pay double (due to the duplicate hardware needed) for such a feature. Some just used it, at our suggestion, as a tool for migrating their workloads from one region or cloud provider to another. The recent GCP and AWS us-east-1 outages vindicated the paranoid among our customers who refused to accept that “cloud regions pretty much never go down.”

Customers who had replication enabled for their namespaces were able to fail over to the other region or cloud, and their applications continued executing as if nothing had happened. We discovered a few misses on our side and had to fail over some namespaces manually, with a longer delay than we expected. The important part is that replicated namespaces continued running after failover. We saw a major spike in customers setting up replication in the days after the AWS us-east-1 outage. One customer was in the process of migrating their namespace from AWS to GCP during GCP’s global outage. They weren’t impacted and didn’t even need to fail over because their active replica was still in AWS. They were considering keeping the cross-cloud replication running indefinitely after that.

I gave a conceptual talk about replicated namespaces, but the topic probably deserves its own post.

Road ahead

With great opportunities come great responsibility and pressure to execute and realize those opportunities. We still have to strike the right balance between running a highly reliable service and investing in new functionality. It’s a deeply humbling experience to see that some of the world’s top companies — household names with tens or even hundreds of millions of users — take an all-in dependency on Temporal Cloud. This leaves no room for hubris, complacency, or sloppiness. We have to keep pushing the reliability and quality bar higher without hampering further development of the product.

I don’t believe there’s a general recipe for how to grow an organization, be it engineering, R&D, or the whole company. We’ll have to navigate our own path — growing sustainably while preserving what has made us successful so far and learning new ways in parallel. It’s exciting and somewhat dizzying at the same time. Yet I feel we are still only getting started.

2025

Sergey Bykov — Wed, 12 Nov 2025 21:56:41 +0000

Part 1

Belated update. Yes, it’s been five years, can’t believe it myself. What’s the “delta” for the last three years that flew by too fast?
Certain things haven’t changed much. We are still under the “dual mandate” — OSS server, SDKs (clients), CLI, and a whole bunch of other peripheral software, plus a cloud service where we charge customers for running and managing the invisible infrastructure so that they don’t have to. My focus continues to be primarily on the cloud side.

At the same time, obviously, everything has changed — some things even multiple times. We went through COVID with its obligatory work-from-home setup, only for many companies to start imposing, some more gradually than others, return-to-office policies. I interviewed a number of candidates recently who moved away from major tech hubs during COVID and had to leave their jobs because of the RTO push. We are still fully remote.

Most of the Big Tech companies went through rounds of mass layoffs — a tectonic shift from the previous 20 or so years of competing for talent and outbidding each other in offers. Startups suddenly became much more attractive for Big Tech employees who were previously reluctant to take the risk of leaving their well-paid jobs. At the same time, startup founders faced the funding drought starting in late 2021 and early 2022 caused by interest rate changes. Many had to close or fire-sell their ventures where just a couple of years earlier, cheap money seemed unlimited.

Product

Developers choose Temporal for its programming model. They experience it in a language of their choice via Temporal SDKs. We started with two languages. Now we support seven: Go, Java, TypeScript, Python, .NET, PHP, and Ruby. Four of them are built on the same Core SDK written in Rust. No, we still don’t have an official Rust SDK.

We started Temporal Cloud by hosting the OSS Temporal Server with an added layer of security and multi-tenancy. The original value proposition included that, plus general operational concerns such as monitoring, alerting, configuration, upgrades, and scale. We’ve been investing along several dimensions since then and are now running the fifth-generation Cells (Temporal Cloud “clusters”).

Building a custom storage layer between the server and database to absorb reads and coalesce writes was one of the first bold undertakings. Rolling it out to production over the course of 2023 gave us a significant increase in reliability, performance, and scalability compared to the vanilla OSS server. Another major investment was making it possible to incrementally add multiple databases to a running server. With these improvements, the scenario I mentioned in one of my previous posts, a major customer needing to increase their already outsized traffic level up to 10x for a day, became routine. At the end of 2021, a day like that was a big deal for both companies, with teams of engineers monitoring the system, communicating live, and taking action. The subsequent occurrences became increasingly “boring” and turned into non-events.

On the authentication/authorization dimension, we went from initially supporting only mTLS and Google SSO to adding API keys, service accounts, SAML, SCIM, and a bunch of other features critical for enterprise — and not only enterprise — customers.

We started with prospective cloud customers filling out a form, getting contacted by our sales team to complete the paperwork, and then creating an account on their behalf. Embarrassing. Now, we have a complete self-signup process that guides prospective customers along the path, with a full PLG motion behind it. When we opened up Temporal Cloud to the world, we were missing a number of table-stakes features. At the time, I called the bar we had to meet a “reasonable cloud service.” I believe we passed this milestone 12–18 months ago.

I like that we don’t play licensing games (our OSS is under the MIT license) and instead extend and enhance it with proprietary features to differentiate our cloud offering.

We launched Temporal Cloud in 2022 with support for AWS only. We added GCP in 2025 and are working on bringing in Azure, the last of the big three providers. Even though support for Kubernetes clusters across them is similar, most of the integration effort goes into their disparate security and resource hierarchy models, differences in networking, and subtle behavioral differences in their seemingly compatible APIs — for example, GCS vs. S3. Recently, we’ve been chasing GCP load balancers mysteriously ghosting a fraction of the connections. Support for hosted Elasticsearch is another headache — only AWS has it, but in the form of OpenSearch, their fork of ES from before Elastic changed its license.

AI

The agentic AI “storm” turned into a sudden tailwind for Temporal. The very nature of such applications — being stateful, depending on a significant number of semi-reliable API calls to external services, and taking seconds to minutes to execute — made code-first Durable Execution a compelling programming model for this fast-moving, massive herd. While there are still some rough edges for AI use cases in the near term (such as payload and history size limits and required determinism of workflow code), the immediate benefits — high-velocity development of much more reliable code in the language of your choice, guaranteed scalability, and unparalleled visibility into execution for debugging — keep bringing AI-focused companies to Temporal. More traditional businesses that are scrambling to integrate AI into their systems do the same. I was told that as of late 2024, out of the top 20 AI companies, only two were aware of Temporal — and now, 16 of them already run Temporal-based apps.

Nexus

This year we launched the initial version of Nexus, an open-standard-based protocol for APIs that may take arbitrarily long to complete. I think of it as a great frontend layer for Durable Execution. But the protocol itself is implementation-agnostic. One could implement it using more traditional tools and approaches, for example, within the paradigm of event-driven architecture. The idea was conceived in the early days of Temporal. We started talking about it publicly in 2022, only to do nothing for another year due to other priorities.

We believe that Nexus is an immense opportunity to integrate systems and services in a new, powerful way. Nexus deserves a dedicated post, and I’m contemplating a conference talk about how the combination of Durable Execution and Nexus could define a major evolution of the Microservice Architecture. I understand this is a very bold statement, but sometimes you have to shoot for the Moon.

(Continued in Part 2)

Building Durable Cloud Control Systems with Temporal

Sergey Bykov — Sat, 09 Aug 2025 00:47:09 +0000

In today’s world of managed cloud services, delivering exceptional user experiences often requires rethinking traditional architecture and operational strategies. At Temporal, we faced this challenge head-on, navigating complex decisions about tenancy models, resource management, and durable execution to build a reliable, scalable cloud service. This post explores our approach and the lessons we learned while creating Temporal Cloud.

The Case for Managed Cloud Services

Managed services have become the default for delivering hosted solutions to customers. Whether it’s a database, queueing system, or another server-side technology, hosting a service not only provides a better user experience but also opens doors for monetization, especially for open-source projects. The challenge is how to do it effectively while maintaining reliability and scalability.

One of the first decisions we made was about tenancy models. Should we pursue single-tenancy — provisioning dedicated clusters for each customer — or opt for multi-tenancy, which allows multiple customers to share the same resources? While single-tenancy offers simplicity and isolation, its inefficiencies quickly become apparent. Customers end up paying for unused capacity, and providers shoulder higher operational costs. Multi-tenancy, though harder to implement, emerged as the clear winner. It optimizes resource usage, allows customers to pay for actual usage, and creates shared headroom for handling traffic spikes.

Data Plane vs. Control Plane: Defining Responsibilities

Architecting a managed service in terms of the data plane and control plane is an industry best practice that we followed, clearly defining and implementing their distinct roles within our cloud architecture.

Data Plane: This is where the actual work happens — processing transactions, executing workflows, and handling customer data. It must maintain high availability, low latency, and resilience to failures. For Temporal Cloud, we adopted a cell-based architecture to isolate resources and minimize the blast radius of potential failures.
Control Plane: This acts as the brain of the system, managing resources, provisioning namespaces, and handling configurations. While its performance is less critical than the data plane, reliability here still matters for customer experience. For instance, provisioning a namespace may not be urgent, but delays or errors in this process can frustrate users.

Implementing the Data Plane: A Cell-Based Architecture

For the data plane, we applied a cell-based architecture to achieve strong isolation and scalability. Each cell operates as a self-contained unit with its own AWS account, VPC, EKS cluster, and supporting infrastructure. While this approach is framed within the context of AWS, we have applied the same principles to Google Cloud Platform (GCP), leveraging its equivalent primitives to ensure consistency and reliability across cloud providers. This approach ensures that failures or updates in one cell do not impact others, reducing the risk of cascading outages.

Each cell in Temporal Cloud includes:

Compute Pods: Running Temporal services and infrastructure tools for observability, ingress management, and certificate handling.
Databases: Both primary databases and Elasticsearch for enhanced visibility.
Additional Components: Load balancers, private connectivity endpoints, and other supporting infrastructure that ensures smooth operation and integration across environments. Currently, Temporal Cloud operates across 14 AWS regions, and we’ve also added support for GCP. This architecture allows us to meet the diverse needs of our customers while maintaining reliability at scale.

Durable Execution: The Foundation of the Control Plane

Building the control plane presented its own set of challenges, particularly around reliability and maintainability. Control plane tasks, such as provisioning namespaces or rolling out updates, involve complex long-running processes with many interdependent steps. Writing this logic as traditional, ad-hoc code often leads to brittle systems that are hard to debug and evolve.

This is where Temporal’s durable execution model shines. Designed based on experience with earlier systems like AWS Simple Workflow Service and Azure Durable Functions, Temporal’s approach separates business logic from state management and failure handling. Developers can write workflows as straightforward, happy-path code without worrying about retries, error handling, or state persistence. The system automatically manages these concerns, allowing workflows to seamlessly recover from failures.

Namespace Provisioning: A Real-World Example

Consider the process of creating a new namespace in Temporal Cloud. When a user clicks “Create Namespace” on the web interface, the control plane orchestrates a series of tasks:

Selecting a suitable cell within the chosen region.
Creating database records and roles.
Generating and provisioning mTLS certificates.
Configuring ingress routes and verifying connectivity. Each step involves external API calls, DNS propagation, and other potential points of failure.

Without durable execution, managing retries, backoffs, and state persistence would result in a tangle of brittle code. With Temporal, these tasks are encapsulated in workflows, which transparently handle retries and maintain state across failures. Developers can focus on the high-level logic, confident that the system will handle the edge cases.

Rolling Upgrades: Ensuring Safe Deployments

Another common control plane scenario is rolling out updates to the Temporal Cloud fleet. Our deployment strategy involves organizing cells into deployment rings, progressing from pre-production environments to customer-facing cells with increasing priority of traffic.

The rollout process is carefully staged:

Ring 0: Synthetic traffic only, no customer impact. Changes are monitored here for at least a week.
Ring 1: Low-priority traffic namespaces, allowing for additional testing with minimal risk.
Higher Rings: Gradually expanding to critical, high-priority traffic customers. Within each ring, updates are applied in batches, with pauses between batches to observe for potential issues like memory leaks or race conditions. Temporal workflows handle this process, ensuring that even long-running deployments (which can span weeks) are resilient to failures or restarts.

Entity Workflows: A Powerful Pattern

Temporal’s durable execution also enables powerful patterns like entity workflows. These are workflows tied to specific resources, such as cells or namespaces, providing a natural way to model state and operations. For example, each cell in Temporal Cloud has an entity workflow that manages its lifecycle, from provisioning to upgrades. This approach ensures consistency and simplifies concurrency control.

Developer Happiness and Productivity

One of the biggest benefits of Temporal’s approach is the impact on developer experience. By eliminating the need to write boilerplate code for retries, backoffs, and state management, developers can focus on delivering business value. Temporal’s built-in tools for observing and debugging workflows further enhance productivity, making it easier to understand and troubleshoot complex systems.

Happy developers are productive developers, and Temporal’s approach fosters this by reducing the cognitive load and frustration associated with traditional workflow coding.

Why Durable Execution Matters

Durable execution is more than a technical innovation; it’s a paradigm shift for building cloud-native systems. By decoupling business logic from state management and failure handling, Temporal empowers developers to build reliable, scalable systems with less effort. Whether you’re managing control planes, provisioning resources, orchestrating complex workflows, performing money transfers, training AI models, or processing social media posts, this approach delivers clear benefits.

At Temporal, we’ve seen firsthand how durable execution transforms the development process, enabling us to deliver a robust managed service that scales with our customers’ needs.

Ready to Transform Your Control Plane?

Temporal isn’t just a tool for building cloud systems; it’s a better way to think about workflows and application architecture. If you’re building or planning a managed cloud service, consider how durable execution can simplify your journey and unlock new possibilities. For more insights into our approach, check out my full talk at QCon.

Why Top Developers Prioritize Failure Management

Sergey Bykov — Sat, 09 Aug 2025 00:35:05 +0000

There’s a saying: “Amateurs study tactics, while professionals study logistics.” In software, this translates to: “Amateurs focus on algorithms, while professionals focus on failures.”

At J on the Beach, I took time in my talk to expand on this saying and explain that real-world systems don’t just need code that works on the “happy path” — they need a safety net for when things go wrong.

Modern software development has layers of complexity. You’re not just writing code; you’re connecting systems across time and space, handling data that doesn’t sleep, and ensuring flawless performance at scale. What sets top developers apart is how they manage failures. Building resilience focuses on ensuring reliability when things inevitably go wrong, not just maintaining uptime.

In this post, we’ll walk through three common approaches to handling failures in software, each with its own strengths and weaknesses. Then we’ll introduce Temporal’s approach, workflow-as-code, which makes it easier to build reliability into your systems from day one.

Three Ways to Handle Failure in Your Software

Failures are inevitable in your distributed systems. When a network link fails, a server times out, or a service crashes, systems need strategies to respond properly and ensure that your operations remain reliable.

Below, we’ll explore three common approaches to coordination between systems — Remote Procedure Calls (RPCs), persistent queues, and workflows — and their relationship to failure management.

1. Request-Response (RPC)

The request-response, or RPC model, is a classic approach. A client makes a request, the server processes it, and sends back a response. In the best-case scenario — the “happy path” — everything works smoothly. Imagine a money transfer request: one service debits the sender while another credits the receiver. If all goes as planned, the transfer completes with no issues.

Pros of the RPC Model

Simplicity: The direct client-server connection makes this model easy to implement for straightforward workflows.
Efficiency on the “happy path”: When things go smoothly, RPC provides fast, efficient responses and low latency.

Cons of the RPC Model

Limited resilience for partial failures: If the client’s request is successful, but a response isn’t received, or a step in the process fails, RPC often requires extensive error-handling code on the client side.
Heavy client burden: Clients must handle errors, recovery, and retries, complicating systems as they scale.
The RPC model works well for simple, synchronous tasks. However, for resilience, it falls short by placing the onus on developers of the RPCs and those consuming them to manage every failure scenario — and this is no trivial matter.

2. Persistent Queues

Persistent queues add a degree of flexibility by decoupling the client from the server. Messages are placed in a queue, and the system processes them asynchronously. Queues help distribute workloads: they support automatic retries and asynchronous processing, which can smooth out demand spikes.

Pros of Persistent Queues

Automatic retries: Persistent queues often support automatic retries, attempting tasks multiple times if they initially fail.
Load distribution: Queues smooth processing under heavy loads, distributing requests over time, to improve system reliability.
Producer-consumer separation: Decoupling producers and consumers allow the queue to function independently, improving fault tolerance.

Cons of Persistent Queues

Loss of ordering: Since queues process messages independently, tasks may execute out of order, causing unexpected issues for dependent operations.
Dead-letter queues: Tasks that continuously fail may require a separate “dead-letter” queue, adding complexity and, typically, manual intervention.
Limited visibility into status: Visibility becomes even more challenging when you have systems that use multiple queues, requiring additional tooling and infrastructure.
Queues work well when you need flexibility and decoupling, but they lack the control and visibility needed for comprehensive failure management.

3. Workflows

Workflows provide a robust solution for orchestrating complex processes across distributed systems. Unlike RPC or queue-based models, workflows manage retries, state, and error handling automatically, making them ideal for long-running or multi-step processes.

Pros of Workflows

Built-in resilience: Workflows handle retries, recovery, and compensation steps automatically, reducing the need for custom error-handling code.
Support for long-running processes: Workflows accommodate processes that span minutes, hours, or even days, making them well-suited for complex tasks.
Enhanced visibility: Workflow systems enable real-time tracking and querying, so both clients and developers can see exactly where each process stands.

Cons of Workflows

Infrastructure requirements: Workflows require a solid infrastructure to manage states, retries, and tracking, which some teams may lack.
Setup complexity: Workflow systems can be complex to set up, especially when building custom solutions to manage workflows.
For complex processes that demand reliability and transparency, workflows provide the most comprehensive solution, though they require dedicated infrastructure to deploy effectively.

Resilience Without Extra Overhead

At Temporal, we addressed these challenges by designing a platform that handles resilience, error handling, and state management so you don’t have to.

With Temporal, you write workflows as code - no extra XML, JSON, or YAML definition of workflow logic that is difficult to understand and debug down the line. Define your steps in regular code, and Temporal does the rest, managing retries, maintaining state, and ensuring that your workflows are reliable and simple to create.

Companies like ANZ Bank, one of the largest banks in the Asia-Pacific region, rely on Temporal to strengthen the resilience and reliability of critical financial processes. With Temporal, ANZ orchestrates and manages complex operations across distributed systems, ensuring tasks are retried automatically, failures are handled, and long-running processes are tracked seamlessly. This has enabled ANZ to boost system reliability, reduce operational complexity, and uphold strict compliance standards in their high-stakes FinServ environment.

Failure Management Is a Strategy, Not a Setback

Any complex system will encounter failures. But how you handle those failures makes all the difference. For developers, focusing on failure management from the start distinguished exceptional teams from the average. Building resilience into your system sets your project up for long-term success.

Time-Travel Debugging Production Code

Loren 🤓 — Tue, 08 Aug 2023 07:03:31 +0000

In this post, I’ll give an overview of time travel debugging (what it is, its history, how it’s implemented) and show how it relates to debugging your production code.

Normally, when we use debuggers, we set a breakpoint on a line of code, we run our code, execution pauses on our breakpoint, we look at values of variables and maybe the call stack, and then we manually step forward through our code's execution. In time-travel debugging, also known as reverse debugging, we can step backward as well as forward. This is powerful because debugging is an exercise in figuring out what happened: traditional debuggers are good at telling you what your program is doing right now, whereas time-travel debuggers let you see what happened. You can wind back to any line of code that executed and see the full program state at any point in your program’s history.

History and current state

It all started with Smalltalk-76, developed in 1976 at Xerox PARC. (Everything started at PARC 😄.) It had the ability to retrospectively inspect checkpointed places in execution. Around 1980, MIT added a "retrograde motion" command to its DDT debugger, which gave a limited ability to move backward through execution. In a 1995 paper, MIT researchers released ZStep 95, the first true reverse debugger, which recorded all operations as they were performed and supported stepping backward, reverting the system to the previous state. However, it was a research tool and not widely adopted outside academia.

ODB, the Omniscient Debugger, was a Java reverse debugger that was introduced in 2003, marking the first instance of time-travel debugging in a widely used programming language. GDB (perhaps the most well-known command-line debugger, used mostly with C/C++) added it in 2009.

Now, time-travel debugging is available for many languages, platforms, and IDEs, including:

Replay for JavaScript in Chrome, Firefox, and Node, and Wallaby for tests in Node
WinDbg for Windows applications
rr for C, C++, Rust, Go, and others on Linux
Undo for C, C++, Java, Kotlin, Rust, and Go on Linux
Various extensions (often rr- or Undo-based) for Visual Studio, VS Code, JetBrains IDEs, Emacs, etc.

Implementation techniques

There are three main approaches to implementing time-travel debugging:

Record & Replay: Record all non-deterministic inputs to a program during its execution. Then, during the debug phase, the program can be deterministically replayed using the recorded inputs in order to reconstruct any prior state.
Snapshotting: Periodically take snapshots of a program's entire state. During debugging, the program can be rolled back to these saved states. This method can be memory-intensive because it involves storing the entire state of the program at multiple points in time.
Instrumentation: Add extra code to the program that logs changes in its state. This extra code allows the debugger to step the program backwards by reverting changes. However, this approach can significantly slow down the program's execution.

rr uses the first (the rr name stands for Record and Replay), as does Replay. WinDbg uses the first two, and Undo uses all three (see how it differs from rr).

Time-traveling in production

Traditionally, running a debugger in prod doesn't make much sense. Sure, we could SSH into a prod machine and start the process handling requests with a debugger and a breakpoint, but once we hit the breakpoint, we're delaying responses to all current requests and unable to respond to new requests. Also, debugging non-trivial issues is an iterative process: we get a clue, we keep looking and find more clues; discovery of each clue is typically rerunning the program and reproducing the failure. So, instead of debugging in production, what we do is replicate on our dev machine whatever issue we're investigating and use a debugger locally (or, more often, add log statements 😄), and re-run as many times as required to figure it out. Replicating takes time (and in some cases a lot of time, and in some cases infinite time), so it would be really useful if we didn't have to.

While running traditional debuggers doesn't make sense, time-travel debuggers can record a process execution on one machine and replay it on another machine. So we can record (or snapshot or instrument) production and replay it on our dev machine for debugging (depending on the tool, our machine may need to have the same CPU instruction set as prod). However, the recording step generally doesn't make sense to use in prod given the high amount of overhead—if we set up recording and then have to use ten times as many servers to handle the same load, whoever pays our AWS bill will not be happy 😁.

But there are a couple scenarios in which it does make sense:

Undo only slows down execution 2–5x, so while we don't want to leave it on just in case, we can turn it on temporarily on a subset of prod processes for hard-to-repro bugs until we have captured the bug happening, and then we turn it off.
When we're already recording the execution of a program in the normal course of operation.

The rest of this post is about #2, which is a way of running programs called durable execution.

Durable execution

What's that?

First, a brief backstory. After Amazon (one of the first large adopters of microservices) decided that using message queues to communicate between services was not the way to go (hear the story first-hand here), they started using orchestration. And once they realized defining orchestration logic in YAML/JSON wasn't a good developer experience, they created AWS Simple Workfow Service to define logic in code. This technique of backing code by an orchestration engine is called durable execution, and it spread to Azure Durable Functions, Cadence (used at Uber for > 1,000 services), and Temporal (used by Stripe, Netflix, Datadog, Snap, Coinbase, and many more).

Durable execution runs code durably—recording each step in a database, so that when anything fails, it can be retried from the same step. The machine running the function can even lose power before it gets to line 10, and another process is guaranteed to pick up executing at line 10, with all variables and threads intact.¹ It does this with a form of record & replay: all input from the outside is recorded, so when the second process picks up the partially-executed function, it can replay the code (in a side-effect–free manner) with the recorded input in order to get the code into the right state by line 10.

Durable execution's flavor of record & replay doesn't use high-overhead methods like software JIT binary translation, snapshotting, or instrumentation. It also doesn't require special hardware. It does require one constraint: durable code must be deterministic (i.e., given the same input, it must take the same code path). So it can't do things that might have different results at different times, like use the network or disk. However, it can call other functions that are run normally ("volatile functions", as we like to call them 😄), and while each step of those functions isn't persisted, the functions are automatically retried on transient failures (like a service being down).

Only the steps that require interacting with the outside world (like calling a volatile function, or calling sleep('30 days'), which stores a timer in the database) are persisted. Their results are also persisted, so that when you replay the durable function that died on line 10, if it previously called the volatile function on line 5 that returned "foo", during replay, "foo" will immediately be returned (instead of the volatile function getting called again). While yes, it adds latency to be saving things to the database, Temporal supports extremely high throughput (tested up to a million recorded steps per second). And in addition to function recoverability and automatic retries, it comes with many more benefits, including extraordinary visibility into and debuggability of production.

Debugging prod

With durable execution, we can read through the steps that every single durable function took in production. We can also download the execution’s history, checkout the version of the code that's running in prod, and pass the file to a replayer (Temporal has runtimes for Go, Java, JavaScript, Python, .NET, and PHP) so we can see in a debugger exactly what the code did during that production function execution. Read this post or watch this video to see an example in VS Code.²

Being able to debug any past production code is a huge step up from the other option (finding a bug, trying to repro locally, failing, turning on Undo recording in prod until it happens again, turning it off, then debugging locally). It's also a (sometimes necessary) step up distributed tracing.

I hope you found this post interesting! If you'd like to learn more about durable execution, I recommend reading:

and watching:

Thanks to Greg Law, Jason Laster, Chad Retz, and Fitz for reviewing drafts of this post.

Technically, it doesn't have line-by-line granularity. It only records certain steps that the code takes—read on for more info ☺️. ↩
The astute reader may note that our extension uses the default VS Code debugger, which doesn’t have a back button 😄. I transitioned from talking about TTD to methods of debugging production code via recording, so while Temporal doesn’t have TTD yet, it does record all the non-deterministic inputs to the program and is able to replay execution, so it’s definitely possible to implement. Upvote this issue or comment if you have thoughts on implementation! ↩

Actors and Workflows: Building a Customer Loyalty Program with Temporal

Fitz — Thu, 03 Aug 2023 17:28:24 +0000

This post is technically a followup of another post. You don't need to read that one to make sense of this one, but it might give some useful background.

That post talked through how the Actor Model can be implemented using "Workflows" (on https://github.com/temporalio/temporal), even though these two concepts don't immediately appear compatible.

Here, I dive into a concrete example: a Workflow representing a customer's loyalty status.

If you want to skip the prose and just jump right into the code, you can find it all in this GitHub repository, with implementations in Go, Java, and Python.

Actor Model Refresher

As formally defined, Actors must be able to do three things:

Send and receive messages
Create new Actors
Maintain state

Exact implementation details vary depending on what framework, library, or tools you're using, but the biggest challenge is having some kind of software artifact running somewhere that can handle these things.

That's where most Actor frameworks come in to help: providing both the programming model and the runtime environment for being able to build an Actor-based application in a highly distributed, concurrent, and scalable way.

Temporal differs here in that it’s general-purpose, rather than specific to one model or system design pattern. With Workflows, you define a function that Temporal will ensure runs to completion (or reliably runs forever, if the function doesn’t return).

I recognize that statement is both rather bold and also so generic as to be hard to disprove. So, let's look at a concrete example.

Loyal Customers

Many consumer businesses have some kind of loyalty program. Buy 10 items, get the 11th free! Fly 10,000 miles, get free access to the airport lounge! Earn one million points over the lifetime of your account, earn a gold star!

At the highest level, the application's logic isn't complex: Each customer has an integer counter that's incremented after the customer does certain things (e.g., buy something, or take a trip). When that counter crosses different thresholds, new rewards are unlocked. And, although we may not like it, customers can always close their accounts.

When we create the diagram for the app, it might look like this:

In terms of the Actor Model, two of the three requirements are on display:

Send and receive messages: A customer can send either an "earn points" message or a "try to use reward" message.
Create new Actors: ??? (This is the Actor requirement not apparent in this application, but we'll see later how it can be incorporated.)
Maintain state: A customer loyalty account needs to maintain the points counter and which rewards are unlocked (or be able to look up this information based on the points value).

Requirement #2, the ability to create other Actors, isn't immediately obvious here, but it isn't too far out of reach. We could define in this example application that one of the rewards for earning enough points is the ability to gift status to someone else, inviting them (i.e., creating their account) to the program if they aren't already a member.

If our goal is to create a demo application for the Actor Model (as it is in this post), then there's actually one other thing missing: the ability for a customer (or rather, their loyalty account) to send messages. For that, we could also declare that customers with enough points can gift points or status levels (i.e., which rewards are unlocked) to their guests. Then they can send messages, too!

Reworking the previous diagram to be more befitting of a full "Actor," we'd get the following:

And, as for the exact implementation details, read on!

Loyal (Temporal!) Customers

Imagine being able to write the customer loyalty program above in just a function or two. Conceptually, that's not hard. In pseudocode, that might look like the following:

INVITE_REWARD_MINIMUM_POINTS = 1000

function CustomerLoyaltyAccount:
    account_canceled = false
    points = 0

    while !account_canceled:
        message = receive_message()
        switch message.type:
            case 'cancel':
                account_canceled = true
            case 'add_points':
                fallthrough
            case 'gift_points':
                points += message.value
            case 'invite_guest'
                if points >= INVITE_REWARD_MINIMUM_POINTS:
                    spawn(new CustomerLoyaltyAccount())

But there are a few crucial details that are, well, rather undefined in this pseudo-function. Specifically:

What's receive_message() doing? How is it receiving messages?
Similarly, what's spawn(new CustomerLoyaltyAccount()) doing?
And most importantly, where is this function running? What happens if that runtime crashes or the function otherwise stops running?

Each of these maps to core Temporal features that we can implement in an example Workflow:

Data can be sent to Workflows via Signals
Workflows can create new Workflow instances
As long as there are Workers running somewhere that can communicate with the Temporal Server, then if the Worker running the function dies, the function will continue running on another (you know, kind of Temporal's main benefit)

Customers Go Loyal

Let's build this up in Go. If you are more comfortable with other languages, I've also written the same Workflow in Python and Java. While the languages are different, most of the same concepts and patterns should carry over.

(For brevity in the body of this blog post, I'll in most cases omit error handling but include it when non-trivial and relevant.)

First, we write the skeleton of a Workflow and an Activity. For some of the milestones in a customer's lifecycle, it'd be nice to send them some kind of notification. In a real application, you'd call out to SendGrid, Mailchimp, Constant Contact, or some other email provider, but for simplicity's sake, I'm just logging out the details. This initial Workflow does just that: if it's a new customer, send a welcome email, but otherwise move on.

func CustomerLoyaltyWorkflow(ctx workflow.Context, customer CustomerInfo, newCustomer bool) error {
    logger := workflow.GetLogger(ctx)
    logger.Info("Loyalty workflow started.", "CustomerInfo", customer)

  var activities Activities

    if newCustomer {
        logger.Info("New customer workflow; sending welcome email.")
        err := workflow.ExecuteActivity(ctx, activities.SendEmail,
            fmt.Sprintf("Welcome, %v, to our loyalty program!", customer.Name)).
            Get(ctx, nil)
        if err != nil {
            logger.Error("Error running SendEmail activity for welcome email.", "Error", err)
        }
    } else {
        logger.Info("Skipping welcome email for non-new customer.")
    }

    // ... [to be added later] ... //

    return nil
}

type Activities struct {
    Client client.Client
}

func (*Activities) SendEmail(ctx context.Context, body string) error {
    logger := activity.GetLogger(ctx)
    logger.Info("Sending email.", "Contents", body)
    return nil
}

Next up, we need to be able to handle messages. This is the primary thing the Workflow (i.e., customer loyalty Actor) does: sit around waiting for new messages to come in.

The following code replaces the // ... [to be added below] ... // line from the previous snippet:

  selector := workflow.NewSelector(ctx)

    // Signal handler for adding points
    selector.AddReceive(workflow.GetSignalChannel(ctx, "addPoints"),
        func(c workflow.ReceiveChannel, _ bool) {
            signalAddPoints(ctx, c, &customer)
        })

    // Signal handler for canceling account
    selector.AddReceive(workflow.GetSignalChannel(ctx, "cancelAccount"),
        func(c workflow.ReceiveChannel, _ bool) {
            signalCancelAccount(ctx, c, &customer)
        })

    // ... [register other Signal handlers here] ... //

  logger.Info("Waiting for new messages")
    for customer.AccountActive {
        selector.Select(ctx)
    }

The Signal handler function for adding points does very little, adding in the given points to the customer's state and then sending an email to the customer with the new value.

As you might imagine, the cancel account handler is very similar, setting the customer.AccountActive flag used above to false and then notifying the customer.

func signalAddPoints(ctx workflow.Context, c workflow.ReceiveChannel, customer *CustomerInfo) {
    logger := workflow.GetLogger(ctx)
    var activities Activities

    var pointsToAdd int
    c.Receive(ctx, &pointsToAdd)

    logger.Info("Adding points to customer account.", "PointsAdded", pointsToAdd)
    customer.LoyaltyPoints += pointsToAdd

    err := workflow.ExecuteActivity(ctx, activities.SendEmail,
        fmt.Sprintf("You've earned more points! You now have %v.", customer.LoyaltyPoints)).
        Get(ctx, nil)
    if err != nil {
        logger.Error("Error running SendEmail activity for added points.", "Error", err)
    }

    // ... [insert logic for unlocking status levels or rewards] ... //
}

All combined, the code so far does three things:

First, it registers the signalAddPoints and signalCancelAccount functions as the handlers for the "addPoints" and "cancelAccount" Signals, respectively.
Then, it blocks forward progress on the Workflow, via selector.Select(ctx), until a registered Signal comes in. Unless that Signal is "cancelAccount," the Workflow will keep looping on this select.
I've chosen for this application to not fail the Workflow when an email fails to send. This keeps the Workflow representing the customer's loyalty account active and running even in spite of external system failure.
- For that, you'll want to set an appropriate retry policy to ensure that the Workflow doesn't completely block on email failures, for example by setting the MaximumAttempts to a reasonably low number like 10.

Already this gives us most of the application. We have a function that runs perpetually, thanks to Temporal, and can receive two different kinds of messages, both of which modify the state of the Workflow with one that also results in the Workflow finishing.

What remains is a couple of more Temporal-specific considerations.

Long-Lived Customers

In my last post, I spilled many words on the topic of "Continue-As-New." If you didn't—or don't want to!—read those words, the gist is this: at some point, a Workflow's history may get unwieldily big; Continue-As-New resets it.

For this customer loyalty example Workflow, the far-and-away biggest contributor to the Event History is the number of events, not the size. With the addPoints Signal only taking a single integer argument and the cancelAccount Signal taking none, the combined contribution to the size of the history is minimal.

A Signal with only a single integer parameter will, by itself, contribute one Event and about 500 bytes to the History, even with very large values. And so, how many of these Signals would be required to hit either the size or length limits?

If nothing else happened but addPoints Signals, it'd take 51,200 of them to reach the length limit, but 50 * 1024 * 1024 / 500 or 104,857.6 to reach the size limit. Knowing that many of these Signals will result in the SendEmail Activity running, and each Activity contributes a handful of (small) events to the history, this Workflow will hit the History length limit well before the size limit.

So, let's add a check for that into our Workflow loop:

    const eventsThreshold = 10_000
    // ... snip ...

    info := workflow.GetInfo(ctx)

    logger.Info("Waiting for new messages")
    for customer.AccountActive && info.GetCurrentHistoryLength() < eventsThreshold {
        selector.Select(ctx)
    }

Finally, trigger Continue-As-New as needed, draining any pending signals:

    if customer.AccountActive {
        logger.Info("Account still active, but hit continue-as-new threshold.")
        // Drain signals before continuing-as-new
        for selector.HasPending() {
            selector.Select(ctx)
        }
        return workflow.NewContinueAsNewError(ctx, CustomerLoyaltyWorkflow, customer, false)
    }

My previous post on this topic explained in a little more detail about why it's necessary to drain signals before continuing-as-new. To briefly recap, Continue-As-New finishes the current Workflow run and starts a new instance of the Workflow regardless of any pending Signals. If we don't drain (and handle!) Signals before calling workflow.NewContinueAsNewError (or workflow.continue_as_new in Python, or Workflow.continueAsNew in Java), those pending Signals will be forever lost.

The last major thing this Workflow needs to make it a true, stage-worthy Actor is the ability to create others.

Spawning New Customers

While Temporal has support for Parent/Child relationships between Workflows, in this customer loyalty application, the only thing we need is the ability to send a message from one to the other in the case of gifting status or points.

Temporal provides an API in the Client that can do this and create other Workflows all in one call, called Signal-with-Start. Since this is only available in the Client, not from a Workflow, we'll need to do this in an Activity.

First, I'm setting the ID Reuse Policy to REJECT. This is in some ways a "business logic" kind of decision, where I'm declaring that once a customer's account is closed, it can't be re-invited. (Note that after a namespace's retention period has passed, IDs from closed Workflows can be reused regardless of this policy, and so in a real-life production version of this app, you'd want to have this check an external source for customer account statuses.)

func (a *Activities) StartGuestWorkflow(ctx context.Context, guest CustomerInfo) error {
    // ...
    workflowOptions := client.StartWorkflowOptions{
        TaskQueue:             TaskQueue,
        WorkflowIDReusePolicy: enums.WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE,
    }

Then, we can call Client.SignalWithStartWorkflow:

    logger.Info("Starting and signaling guest workflow.", "GuestID", guest.CustomerID)
    _, err := a.Client.SignalWithStartWorkflow(ctx, CustomerWorkflowID(guest.CustomerID),
        SignalEnsureMinimumStatus, guest.StatusLevel.Ordinal,
        workflowOptions, CustomerLoyaltyWorkflow, guest, true)

Note the use of the Client from the Activities receiver struct! I'm making use of something in the way Temporal works in Go: if, when we instantiate and register the Activities in the Worker, we also set this Client, then the same connection will be available within the Activities. This way, we don't have to worry about re-creating the Client.

I'm also ignoring the returned future from SignalWithStartWorkflow via a Go convention of assigning to _; because this "guest" Workflow is expected to run indefinitely long, blocking on its results would prevent the original Workflow from doing anything else. Since the future returned from starting a Workflow is either used for waiting for the Workflow to finish, or getting its IDs (which we already know from the CustomerWorkflowID(guest.CustomerID) call), we can safely ignore it.

But, it's still necessary to handle the error. With the ID Reuse Policy set to REJECT, retrying the resulting error from trying to start a an already-closed Workflow will get us nowhere, and so we should instead send some useful information back to the Workflow:

    target := &serviceerror.WorkflowExecutionAlreadyStarted{}
    if errors.As(err, &target) {
        return GuestAlreadyCanceled, nil
    } else if err != nil {
        return -1, err
    }

    return GuestInvited, nil
}

// ... [Defined at top] ...
type GuestInviteResult int

const (
    GuestInvited GuestInviteResult = iota
    GuestAlreadyCanceled
)

Back in the Workflow, after running this Activity I can then check for that error and notify the customer as appropriate. As before, I'm allowing the Workflow to continue if sending the email failed. But if that SignalWithStartWorkflow call failed for any reason other than the guest's account already existing, I want to make some noise and fail the Workflow—something unusual is likely happening.

var inviteResult GuestInviteResult
err := workflow.ExecuteActivity(ctx, activities.StartGuestWorkflow, guest).
    Get(ctx, &inviteResult)
if err != nil {
    return fmt.Errorf("could not signal-with-start guest/child workflow for guest ID '%v': %w", guestID, err)
}

if inviteResult == GuestAlreadyCanceled {
    emailToSend = "Your guest has canceled!"
} else {
    emailToSend = "Your guest has been invited!"
}

err := workflow.ExecuteActivity(ctx, activities.SendEmail, emailToSend).Get(ctx, nil)

This snippet of code would end up being in a Signal handler for something like an "invite guest" Signal. The handler would also include, as discussed at the top of this post, a check for if the current customer is even allowed to do this action.

Summing it all up

There are a few other things to explore in this app, like catching a cancellation request or looking through the tests, but this post has gotten long enough as it is. 🙂

Hopefully this post serves as a nice "close-to-real-world" example for you of how to build something that looks like an "Actor"—aka, a really, really long running Workflow that can send and receive messages and maintain state without a database—using Temporal.

For more information related to this post and about Temporal, check out the following links:

This post's source code (As of publishing, available in Java, Go, and Python)
Actors & Workflows, Part 1
SignaIWithStart
Documentation
Developer Guide

And the best way to learn Temporal is with our free courses.

Cover image from John Jennings on Unsplash

To Choreograph or Orchestrate your Saga, that is the question.

Emily Fortuna — Wed, 12 Jul 2023 17:46:50 +0000

The saga pattern is a distributed systems design pattern for a task that spans machine or microservice boundaries in which full execution of all steps is necessary. Partial execution is not desirable. A common life example used to explain when the saga pattern is useful is trip planning. If you’re planning on attending Replay, for example, you’d need to book a conference ticket, an airplane ticket, and a hotel. If you fail to acquire one of these things, you’ll miss out on meeting fun people in backend engineering face-to-face.

Below the surface, there are two main ways microservices can talk to one another that make your saga possible: choreography and orchestration.

Choreography

Choreography is analogous to ants in an ant colony. Like ants, each microservice has local knowledge, and shares information about state changes with other services via chemical signals called pheromones–I mean via message passing. Just as an ant trail to food emerges organically from pheromones, the overall behavior of a system as a whole that contains choreographed microservices emerges organically from each microservice’s instructions.

One tenant drilled into every software engineer’s head is the value of decoupling. Choreography embodies this idea and is straightforward to implement as a whole. Choreography can be a popular, easy choice for systems that are incrementally moving from a monolith to a microservices architecture. However, if you have any sort of ordering requirement of tasks, such as ordered steps in your saga, choreography can get unwieldy fairly quickly. Suppose we want to book the plane first so that the hotel can know your flight number and pick you up from the airport. Then we book our conference ticket (maybe there’s a discount with certain hotels). The sequence of messages that each service responded to would look like this:

However, just from looking at each microservice’s individual codebase, it’s difficult to understand the order that the system should have since that ordering is distributed throughout the code. This leads to all sorts of higher level business logic diagrams that need to be kept in sync with the code…but wouldn’t it be better if the code were just easier to read in the first place? It also can be difficult to debug the exact sequence of events that lead to a bug since control flow is not immediately clear. So, unless all of your microservices are truly independent of one another and don’t have any sort of “happens before” logic, consider using orchestration instead.

Orchestration

Orchestration, on the other hand, is like an air traffic control tower directing planes, or microservices. One service, a “super microservice” if you will, functions as the message broker sending messages directly to individual microservices telling them what to do, just like planes wait for permission to take off.

Because orchestration centralizes control flow, debugging and understanding control flow is much simpler. Additionally, since each step doesn’t need to keep track of what “happens before” messages it needs to listen to, the code for individual microservices is much simpler. Orchestration also shines in situations where many services need to interact in a single saga step. The glaring Achilles’ heel of this method is that bane of all distributed systems: the message broker is a single point of failure.

Putting it all together

So to summarize, choreography:

Is decentralized and decoupled
Is good for highly independent microservices
Is “easier” to implement, at least initially
Is an easy choice for converting established monoliths to microservices
Can make control flow unclear
Can be challenging to debug

and orchestration:

Has one service issuing “commands” to execute microservices
Makes control flow easier to understand
Easier to build with greenfield applications
Makes debugging and failure handling clearer
Is “harder” to implement initially, but pays dividends later
Has a single point of failure (the message broker)

The interesting tradeoff between these two approaches is one wants to reach for the light, agile option (choreography) in the early days and avoid over-architecting your project, but counterintuitively, orchestration is often easier to build when one uses it from the start.

So, what does Temporal do?

Temporal uses orchestration under the hood (you won’t have to implement it yourself), but also avoids that crucial drawback of a single point of failure. How is such a thing possible? Internally, Temporal records your program’s progress in a log. If that message broker were to go offline, your entire program’s history will have been saved and so that another machine can start up exactly where your program left off, as if nothing happened. This makes Temporal completely horizontally scalable.

To bring this idea back to the saga pattern, an important component of the saga pattern is driving towards completion of all the steps of the saga. The fact that Temporal ensures no progress will ever be lost means it will pick up exactly where it left off “no matter what”, including failures for an unknown length of time, completing the saga with no extra code or heavy lifting on your part.

Additionally, unlike some orchestration engines, in Temporal, the logic of your workflow is expressed entirely in code, so you don’t have to deal with json. In essence, nothing additional is needed to make a robust, failure resilient application other than the business logic of your application itself.

Conclusion

Choreography and orchestration provide different approaches to coordinating communication between microservices. Choreography is decoupled but can make debugging and control flow difficult to follow. Orchestration is centralized, but results in a single point of failure. Temporal uses orchestration under the covers, but by design safeguards against a single point of failure, allowing you to focus on writing your code with the confidence that it is failure resilient.

25 Key Terms for Speaking Distributed Systems and Temporal (an emoji-based guide)

Emily Fortuna — Thu, 29 Jun 2023 18:55:20 +0000

So you want to keep up with all the cool kids throwing around terms like “multi-cluster replication” but you don’t have time to read several textbooks. This handy quick-reference will give you the framework for following (and participating in!) conversations involving distributed systems or Temporal with ease. At the next dinner party you’ll be able to win friends and influence people with your ability to explain distributed systems succinctly in plain English… because we all know you’re not gonna do it with salad. This guide builds upon itself, with terms requiring no additional context first.

Core Distributed Systems Terms To Know

concurrency ↠

Roughly, the idea of running multiple things at once. Two people eating dinner at the same time (“in parallel”) are eating concurrently. Your operating system context switching between a web browser and IDE is also a form of concurrency.

scalability 📈

The ability for a system, such as a website, to accommodate a growing number of requests or work. You can improve scalability by finding places where work can be executed simultaneously or removing performance bottlenecks.

reliability ✅

The likelihood of a system to run without failure for a period of time. Systems can be made more reliable by reducing single points of failure, and detecting failures quickly.

eventual consistency 🐌

Let’s say you’ve replicated a database to improve reliability (and possibly scalability). Great! Eventual consistency says a change in the data in one location will eventually be updated in every location that the database lives; however, until every location is updated, a read from one of the locations may not yet have the updated value. You, the programmer, need to bear in mind you may not always have the most up-to-date data when working under this model.

strong consistency 💪

The guarantee that a data store will always provide the most up-to-date value.

CAP Theorem 🧢

The rule that you gotta pick two of the three: (strong) consistency, availability, partition tolerance. Any distributed data store can only provide at most any two of these qualities, alas. Availability is defined as every request returns a non-error response. Partition tolerance is the ability for a system to continue to operate despite requests between data store nodes being delayed or dropped. See also strong consistency and eventual consistency.

ACID 🧪

Hardcore-sounding acronym borrowed from databases that stands for atomicity, consistency, isolation, durability. See strong consistency, eventual consistency, and other deets below.

atomicity ⚛️

Executing a sequence of operations all together as if they were a single unit, or not at all.

isolation 📦

Executing a sequence of operations concurrently with another sequence has the same effect as executing each operation sequentially.

durability 🗿

Think long-lasting. Standing the test of time. Persisting—i.e. written to disk, or if you were really hardcore, etched on a stone tablet—at which point it can be looked up even in the face of system failure such as a power outage or crash.

durable execution 🔜

Similar to durability, once a program has started executing, it will continue executing to completion. Persisting every step the program takes so that execution can be continued by another process if the current process dies.

idempotent function 🥪

Scary-sounding word, less scary meaning: a function that has the same observed result when called with the same inputs, whether it is called one time or many times.

A function setting some field foo=3? Idempotent. The function foo += 3? Not idempotent, because the value of foo is dependent on the number of times your function is called. Naive implementations of functions that transfer money or send emails are also not idempotent by default.

deterministic function 🧮

Code that always has the same effect/output when given a particular input. Things that are not deterministic use some external state such as user input, a random number, or stored data. Code that reads or writes to a variable that other code can also modify simultaneously is also not deterministic.

platform 💻

Windows, iOS, Docker, and VMware are all platforms. They’re execution environments that define how programs behave inside them. Temporal is also a platform, which defines that code run with Temporal is failure and timeout resilient. You may see the term platform-level used in relation to failures. Platform-level failures are caused by low-level issues such as network errors or process crashes.

application 〉

The code you write. You may see the term application-level used in relation to failures. Application-level failures are domain-specific failures like “insufficient inventory”, or “user canceled ride request.”

event sourcing 🎤

A design pattern that creates event objects for every state change in a system, and records this sequence of events in a log (or event history). Temporal uses event sourcing “under the hood” to ensure failure resilience.

Temporal-Specific Terms

Temporal ✧

A way to run your code, a service and library that work together, that ensures your code never gets stuck in failure at the application-level. or the platform-level. While libraries like async-retry take care of retry logic for functions that fail, what happens if your code making that library call crashes? Temporal says “we gotchu.” It abstracts away complex concepts around retries, rollbacks, queues, state machines, and timers, so that no matter where the failure happens, we’ll ensure your code keeps running the way you want.

Worker 👷

The process that’s actually doing the work executing all of your Temporal code (the Workflow and Activities). Capitalized here to denote the Temporal-specific concept of a Worker, to differentiate from the generic idea of a worker process.

Workflow 📖

The high-level business logic of your program. Essentially, this is where the logic of your application begins. (Technically execution starts with the Worker, and the Worker runs the Workflow code.) All Workflow logic must be deterministic.

Activity 💾

Components of your Workflow that might fail, like network or file system calls, inventory holds, or credit card charges. The decision around how many Activities your program should have–whether you make a separate Activity for every non-deterministic call or put the entire rest of your program in an Activity (don’t do that)–is generally a function of how you’d like your program to behave when retrying a failure. For example, if a downstream instruction should always grab the very freshest data when retrying, those instructions should be grouped together in a single Activity. If you can retry with the old data, they can be in separate Activities. Since Activities can be retried, they should be idempotent.

Query 🙋

A way to inspect the state of a Workflow. The results are guaranteed to show the most recent state.

Signal 🧑‍🏫

A way to notify or send information to a Workflow. A common use case is notifying a Workflow that the user added items to their shopping cart.

retry 🔄

Generally, re-executing an Activity that has failed. Technically, Workflows can also be retried, but they are far less common, such as a developer attempting to update Workflow code running in production.

Cluster 🏘️

The collection of services and databases that make failure and timeout resilience possible. You might sometimes see this colloquially called the Temporal Server.

History 🗃️

A log of events that happened over the course of execution. This log contains attempts to run Activities, Workflow status changes (started, failed, scheduled, etc), timer events, and external information signaled to the system during the run.

In Closing

Knowing this core set of 25 terms should give you sufficient lay-of-the-land to sling references like ACID and Workflow in conversations with coworkers, friends, and family with ease! Better yet, you now know enough to dive deeper into subdomains of interest. If you’d like to try out these terms in practice, check out our getting started guides, courses, and examples in Go, Java, Python, PHP, and TypeScript.

Tuning Temporal Server request latency on Kubernetes

Rob Holland — Thu, 15 Jun 2023 16:44:17 +0000

Request latency is an important indicator for the performance of Temporal Server. Temporal Cloud can offer reliably low request latencies, thanks to its custom persistence backend and expertly managed Temporal Server infrastructure. In this post, we’ll give you some tips for getting lower and more predictable request latencies, and making more efficient use of your nodes, when deploying a self-hosted Temporal Server on Kubernetes.

When evaluating the performance of a Temporal Server deployment, we begin by looking at metrics for the request latencies your application, or workers, observe when communicating with Temporal Server. In order for the system as a whole to run efficiently and reliably, requests must be handled with consistent, low latencies. Low latencies allow us to get high throughput, and stable latencies avoid unexpected slowdowns in our application and allow us to monitor for performance degradation without triggering false alerts.

For this post, we’ll use the History service as our example, which is the service responsible for handling calls to start a new workflow execution, or to update a workflow’s state (history) as it makes progress. None of these tips are specific to the History service—most of them can be applied to all the Temporal Server services.

The curious case of the unexpected throttling

Generally, Kubernetes deployments will set CPU limits on containers to stop them from being able to consume too much CPU, starving other containers running on the same node. The way this is enforced is using something called CPU throttling. Kubernetes converts the CPU limit you set on the container into a limit on CPU cycles per 1/10th second. If the container tries to use more than this limit, it is “throttled”, which means its execution is delayed. This can have a non-trivial impact on the performance of containers, as it can increase request latency. This is particularly true for requests requiring CPU intensive tasks, such as obtaining locks.

For monitoring the Kubernetes clusters in our Scaling series (first post here) we use the kube-prometheus stack.

In contrast to the 1/10th second used to manage CPU throttling, the Prometheus system uses an interval of 15 seconds or more between scrapes of aggregated CPU metrics. The large difference in intervals between the throttling period and the monitoring scraping interval means that CPU throttling can be occurring even if CPU usage metrics are reporting a long way under 100% usage. For this reason, it’s important to monitor CPU throttling specifically.

Here is an example for the History service:

We can see from the dashboard that although the history pods’ CPU usage is reporting below 60%, it is being throttled.

In kube-prometheus setups, you can use this Prometheus query to check for CPU throttling, adjusting the namespace and workload selectors as appropriate:

sum(
    increase(container_cpu_cfs_throttled_periods_total{job="kubelet", metrics_path="/metrics/cadvisor", container!=""}[$__rate_interval])
    * on(namespace,pod)
    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{namespace="temporal", workload="temporal-history"}
)
/
sum(
    increase(container_cpu_cfs_periods_total{job="kubelet", metrics_path="/metrics/cadvisor", container!=""}[$__rate_interval])
    * on(namespace,pod)
    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{namespace="temporal", workload="temporal-history"}
) > 0

So, how can we fix the throttling? Later we’ll discuss why you should probably stop using CPU limits entirely, but for now, as Temporal Server is written in Go, there is something else we can do to improve latencies.

GOMAXPROCS in Kubernetes

GOMAXPROCS is a runtime setting for Go that controls how many processes it’s allowed to fork to provide concurrent processing. By default, Go will assume that it can fork a process for each core on the machine it’s running on, giving it a high level of concurrency.

On a Kubernetes cluster, however, containers will generally not be allowed to use the majority of the cores on a node, due to CPU limits. This mismatch means that Go will make bad decisions about how many processes to fork, leading to inefficient CPU usage. It will (among other things) have to run garbage collection and other housekeeping tasks on CPU cores that it isn’t able to use for any useful amount of real work. As an example: on our Kubernetes cluster, the nodes have 8 cores, but our history pods are limited to 2 cores. This means they may create up to 8 processes, but across those 8 only be able to use a total of 2 cores' share of cycles in every throttling period. It then becomes easy for the container’s processes to starve each other of allowed CPU cycles.

To fix this, we can let Go know how many cores it’s allowed to use by setting the GOMAXPROCS environment variable to match our CPU limit. Note: GOMAXPROCS must be an integer, so you should set it to the number of whole cores you set in the limit. Let’s see what happens when we set GOMAXPROCS on our deployments:

On the left of the graphs, you can see the performance with the default GOMAXPROCS setting. Towards the right, you can see the results of setting the GOMAXPROCS environment variable to “2”, letting Go know it should only use at most 2 processes. CPU throttling has gone entirely, which has helped make our latency more stable. We can also see that because Go can make better decisions about how many processes to create, our CPU usage has lowered, even though performance has actually improved slightly (request latency has lowered). Here, you can see how the CPU across all Temporal services drops after adjusting GOMAXPROCS:

To help give a better experience out of the box, from release 1.21.0 onwards, Temporal will automatically set GOMAXPROCS to match Kubernetes CPU limits if they are present and the GOMAXPROCS environment variable is not already set. Before that release, you should manually set the GOMAXPROCS environment variable for your Temporal Cluster deployments. Also note that GOMAXPROCS will not automatically be set based on CPU requests, only limits. If you are not using CPU limits, you should set GOMAXPROCS manually to close to (equal or slightly greater) than your CPU request. This allows Go to make good decisions about CPU efficiency, taking your CPU requests into consideration.

Which brings us nicely to our second suggestion…

CPU limits probably do more harm than good

Now that we’ve improved the efficiency of our CPU usage, I’m going to echo the sentiment of Tim Hockin (of Kubernetes fame) and many others and suggest that you stop using CPU limits entirely. CPU requests should be closely monitored to ensure you are requesting a sensible amount of CPU for your containers, so that Kubernetes can make good decisions about how many pods it assigns to a node. This allows containers that are having a CPU burst to make use of any spare CPU on the node. Make sure to monitor node CPU usage as well—frequently running out of CPU on the node tells you that pods are bursting more often than your requests allow for, and you should re-examine their CPU requests.

If you can’t disable limits entirely as they enforce some business requirements (customer isolation for example), then consider dedicating some nodes to the Temporal Cluster and use taints and tolerations to pin the deployments to those nodes. This allows you to remove CPU limits from your Temporal Cluster deployments while leaving them in place for your other workloads.

Avoiding increased latency from re-balancing during Temporal upgrades

Temporal Server’s History service automatically balances history shards across the available history pods, this is what allows Temporal Cluster to scale horizontally. Note: Although we use the term balance here, Temporal does not guarantee that there will be an equal number of shards on each pod. The History service will rebalance shards every time a new history pod is added or removed, and this process can take a while to settle. Depending on the scale of your cluster, this rebalancing can increase the latency for requests, as a shard cannot be written to while it is being reassigned to a new history or pod. The effect of this will vary depending on what percentage of shards each of the pods is responsible for. The fewer pods you have, the greater the effect on latency when they are added/removed.

The latency spike during a rollout can be mitigated in two ways, depending on the number of history pods you have:

If you have more than 10 pods, the best option will be to do rollouts slowly, ideally one pod at a time. You can use low values for maxSurge and maxUnavailable to ensure pods are rotated slowly. Using minReadySeconds, or a startupProbe with initialDelaySeconds, can give Temporal Server time to rebalance as each pod is added.

If you have less than 10 pods, it’s better to rotate pods quickly so that rebalancing can settle quickly. You will see latency spikes for each change, but the overall impact will be lower. You can experiment with the maxSurge and maxUnavailable settings to allow Kubernetes to roll out more pods at the same time. The defaults are 25% for each, which for 4 pods would mean only 1 pod will be rotated at once. Your mileage will vary based on scale and load, but we’ve had good success with 50% for maxSurge/maxUnavailable on low (4 or less) pod counts.

Pull-based monitoring systems such as Prometheus use a discovery mechanism to find pods to scrape for metrics. As there is a delay between a pod being started and Prometheus being aware of it, the pod may not be scraped for a few intervals after starting up. This means metrics can report inaccurate values during a deployment, until all the new pods are being scraped.

For this reason, it’s best to ensure you are not using metrics that are emitted by the History service when evaluating History deployment strategies. Instead, SDK metrics such as StartWorkflowExecution request latency are a good fit here. Frontend metrics can also be useful, as long as the Frontend service is not being rolled out at the same time as the History service.

These same deployment strategies are also useful for the Matching service, which balances task queue partitions across matching pods.

Summary

In this post we’ve discussed CPU throttling, CPU limits, and the effect of rebalancing during Temporal upgrades/rollouts. Hopefully, these tips will help you save some money on resources, by using less CPU, and improve the performance and reliability of your self-hosted Temporal Cluster.

We hope you’ve found this useful, we’d love to discuss it further or answer any questions you might have. Please reach out with any questions or comments on the Community Forum or Slack. My name is Rob Holland, feel free to reach out to me directly on Temporal’s Slack if you like, would love to hear from you. You can also follow us on Twitter if you’d like more of this kind of content.

Scaling Temporal: The Basics

Rob Holland — Thu, 15 Jun 2023 16:42:55 +0000

Scaling your own Temporal Cluster can be a complex subject because there are infinite variations on workload patterns, business goals, and operational goals. So, for this post, we will help make it simple and focus on metrics and terminology that can be used to discuss scaling a Temporal Cluster for any kind of workflow architecture.
By far the simplest way to scale is to use Temporal Cloud. Our custom persistence layer and expertly managed Temporal Clusters can support extreme levels of load, and you pay only for what you use as you grow.
In this post, we'll walk through a process for scaling a self-hosted instance of Temporal Cluster.

Out of the box, our Temporal Cluster is configured with the development-level defaults. We’ll work through some load, measure, scale iterations to move towards a production-level setup, touching on Kubernetes resource management, Temporal shard count configuration, and polling optimization. The process we’ll follow is:

Load: Set or adjust the level of load we want to test with. Normally, we’ll be increasing the load as we improve our configuration.
Measure: Check our monitoring to spot bottlenecks or problem areas under our new level of load.
Scale: Adjust Kubernetes or Temporal configuration to remove bottlenecks, ensuring we have safe headroom for CPU and memory usage. We may also need to adjust node or persistence instance sizes here, either to scale up for more load or scale things down to save costs if we have more headroom than we need.
Repeat

Our Cluster

For our load testing we’ve deployed Temporal on Kubernetes, and we’re using MySQL for the persistence backend. The MySQL instance has 4 CPU cores and 32GB RAM, and each Temporal service (Frontend, History, Matching, and Worker) has 2 pods, with requests for 1 CPU core and 1GB RAM as a starting point. We’re not setting CPU limits for our pods—see our upcoming Temporal on Kubernetes post for more details on why. For monitoring we’ll use Prometheus and Grafana, installed via the kube-prometheus stack, giving us some useful Kubernetes metrics.

Scaling Up

Our goal in this post will be to see what performance we can achieve while keeping our persistence database (MySQL in this case) at or below 80% CPU. Temporal is designed to be horizontally scalable, so it is almost always the case that it can be scaled to the point that the persistence backend becomes the bottleneck.

Load

To create load on a Temporal Cluster, we need to start Workflows and have Workers to run them. To make it easy to set up load tests, we have packaged a simple Workflow and some Activities in the benchmark-workers package. Running a benchmark-worker container will bring up a load test Worker with default Temporal Go SDK settings. The only configuration it needs out of the box is the host and port for the Temporal Frontend service.

To run a benchmark Worker with default settings we can use:

kubectl run benchmark-worker \
    --image ghcr.io/temporalio/benchmark-workers:main \
    --image-pull-policy Always \
    --env "TEMPORAL_GRPC_ENDPOINT=temporal-frontend.temporal:7233"

Once our Workers are running, we need something to start Workflows in a predictable way. The benchmark-workers package includes a runner that starts a configurable number of Workflows in parallel, starting a new execution each time one of the Workflows completes. This gives us a simple dial to increase load, by increasing the number of parallel Workflows that will be running at any given time.

To run a benchmark runner we can use:

kubectl run benchmark-worker \
    --image ghcr.io/temporalio/benchmark-workers:main \
    --image-pull-policy Always \
    --env "TEMPORAL_GRPC_ENDPOINT=temporal-frontend.temporal:7233" \
    --command -- runner -t ExecuteActivity '{ "Count": 3, "Activity": "Echo", "Input": { "Message": "test" } }'

For our load test, we’ll use a deployment rather than kubectl to deploy the Workers and runner. This allows us to easily scale the Workers and collect metrics from them via Prometheus. We’ll use a deployment similar to the example here: github.com/temporalio/benchmark-workers/blob/main/deployment.yaml

For this test we’ll start off with the default runner settings, which will keep 10 parallel Workflows executions active. You can find details of the available configuration options in the README.

Measure

When deciding how to measure system performance under load, the first metric that might come to mind is the number of Workflows completed per second. However, Workflows in Temporal can vary enormously between different use cases, so this turns out to not be a very useful metric. A load test using a Workflow which just runs one Activity might produce a relatively high result compared to a system running a batch processing Workflow which calls hundreds of Activities. For this reason, we use a metric called State Transitions as our measure of performance. State Transitions represent Temporal writing to its persistence backend, which is a reasonable proxy of how much work Temporal itself is doing to ensure your executions are durable. Using State Transitions per second allows us to compare numbers across different workloads. Using Prometheus, you can monitor State Transitions with the query:

sum(rate(state_transition_count_count{namespace="default"}[1m]))

Now we have State Transitions per second as our throughput metric, we need to qualify it with some other metrics for business or operational goals (commonly called Service Level Objectives or SLOs). The values you decide on for a production SLO will vary. To start our load tests, we are going to work on handling a fixed level of load (as opposed to spiky) and expect a StartWorkflowExecution request latency of less than 150ms. If a load test can run within our StartWorkflowExecution latency SLO, we’ll consider that the cluster can handle the load. As we progress we’ll add other SLOs to help us decide if the cluster can be scaled to handle higher load, or to more efficiently handle the current load.

We can add a Prometheus alert to make sure we are meeting our SLO. We’re only concerned about StartWorkflowExecution requests for now, so we filter the operation metric tag to focus on those.

   - alert: TemporalRequestLatencyHigh
     annotations:
       description: Temporal {{ $labels.operation }} request latency is currently {{ $value | humanize }}, outside of SLO 150ms.
       summary: Temporal request latency is too high.
     expr: |
       histogram_quantile(0.95, sum by (le, operation) (rate(temporal_request_latency_bucket{job="benchmark-monitoring",operation="StartWorkflowExecution"}[5m])))
       > 0.150
     for: 5m
     labels:
       namespace: temporal
       severity: critical

Checking our dashboard, we can see that unfortunately our alert is already firing, telling us we’re failing our SLO for request latency.

Scale

Obviously, this is not where we want to leave things, so let’s find out why our request latency is so high. The request we’re concerned with is the StartWorkflowExecution request, which is handled by the History service. Before we dig into where the bottleneck might be, we should introduce one of the main tuning aspects of Temporal performance, History Shards.

Temporal uses shards (partitions) to divide responsibility for a Namespace’s Workflow histories amongst History pods, each of which will manage a set of the shards. Each Workflow history will belong to a single shard, and each shard will be managed by a single History pod. Before a Workflow history can be created or updated, there is a shard lock that must be obtained. This needs to be a very fast operation so that Workflow histories can be created and updated efficiently. Temporal allows you to choose the number of shards to partition across. The larger the shard count, the less lock contention there is, as each shard will own fewer histories, so there will be less waiting to obtain the lock.

We can measure the latency for obtaining the shard lock in Prometheus using:

histogram_quantile(0.95, sum by (le)(rate(lock_latency_bucket[1m])))

Looking at the History dashboard we can see that shard lock latency p95 is nearly 50ms. This is much higher than we’d like. For good performance we’d expect shard lock latency to be less than 5ms, ideally around 1ms. This tells us that we probably have too few shards.

The shard count on our cluster is set to the development default, which is 4. Temporal recommends that small production clusters use 512 shards. To give an idea of scale, it is rare for even large Temporal clusters to go beyond 4,096 shards.

The downside to increasing the shard count is that each shard requires resources to manage. An overly large shard count wastes CPU and Memory on History pods; each shard also has its own task processing queues, which puts extra pressure on the persistence database. One thing to note about shard count in Temporal is that it is the one configuration setting which cannot (currently) be changed after the cluster is built. For this reason it’s very important to do your own load testing or research to determine what a sensible shard count would be, before building a production cluster. In future we hope to make the shard count adjustable. As this is just a test cluster, we can rebuild it with a shard count of 512.

After changing the shard count, the shard lock latency has dropped from around 50ms to around 1ms. That’s a huge improvement!

However, as we mentioned, each shard needs management. Part of the management includes a cache of Workflow histories for that shard. We can see the History pods’ memory usage is rising quickly. If the pods run out of memory, Kubernetes will terminate and restart them (OOMKilled). This causes Temporal to rebalance the shards onto the remaining History pod(s), only to then rebalance again once the new History pod comes up. Each time you make a scaling change, be sure to check that all Temporal pods are still within their CPU and memory requests—pods frequently being restarted is very bad for performance! To fix this, we can bump the memory limits for the History containers. Currently, it is hard to estimate the amount of memory a History pod is going to use because the limits are not set per host, or even in MB, but rather as a number of cache entries to store. There is work to improve this: github.com/temporalio/temporal/issues/2941. For now, we’ll set the History memory limit to 8GB and keep an eye on them—we can always raise it later if we find the pod needs more.

After this change, the History pods are looking good. Now that things are stable, let’s see what impact our changes have had on the State Transitions and our SLO.

State Transitions are up from our starting point of 150/s to 395/s and we’re way below our SLO of 150ms for request latency, staying under 50ms, so that’s great! We’ve completed a load, measure, scale iteration and everything looks stable.

Round two!

Load

After our shard adjustment, we’re stable, so let’s iterate again. We’ll increase the load to 20 parallel workflows.

Checking our SLO dashboard, we can see the State Transitions have risen to 680/s. Our request latency is still fine, let’s bump the load to 30 parallel workflows and see if we get more State Transitions for free.

We can see we did get another raise in State Transitions, although not as dramatic. Time to check dashboards again.

Measure

History CPU is now exceeding its requests at times—we’d like to aim to have some headroom. Ideally, the majority of the time the process should use under 80% of its request, so let’s bump the History pods to 2 cores.

History CPU is looking better now, how about our State Transitions?

We’re doing well! State Transitions are now up to 1,200/s and request latency is back down to 50ms. We’ve got the hang of the History scaling process, so let’s move on to look at another core Temporal sub-system, polling.

Scale

While the History service is concerned with shuttling event histories to and from the persistence backend, the polling system (known as the Matching service) is responsible for matching tasks to your application workers efficiently.

If your Worker replica count and poller configuration are not optimized, then there will be a delay between the time a task is requested and when it is processed. This is known as Schedule-to-Start latency, and this will be our next SLO. We’ll aim for 150ms like we do for our Request Latency SLO.

   - alert: TemporalWorkflowTaskScheduleToStartLatencyHigh
     annotations:
       description: Temporal Workflow Task Schedule to Start latency is currently {{ $value | humanize }}, outside of SLO 150ms.
       summary: Temporal Workflow Task Schedule to Start latency is too high.
     expr: |
       histogram_quantile(0.95, sum by (le) (rate(temporal_workflow_task_schedule_to_start_latency_bucket{namespace="default"}[5m])))
       > 0.150
     for: 5m
     labels:
       namespace: temporal
       severity: critical
   - alert: TemporalActivityScheduleToStartLatencyHigh
     annotations:
       description: Temporal Activity Schedule to Start latency is currently {{ $value | humanize }}, outside of SLO 150ms.
       summary: Temporal Activity Schedule to Start latency is too high.
     expr: |
       histogram_quantile(0.95, sum by (le) (rate(temporal_activity_schedule_to_start_latency_bucket{namespace="default"}[5m])))
       > 0.150
     for: 5m
     labels:
       namespace: temporal
       severity: critical

After adding these alerts, let’s check out the polling dashboard.

So we can see here that our Schedule-to-Start latency for Activities is too high. We’re taking over 150 ms to begin an Activity after it’s been scheduled. The dashboard also shows another polling related metric which we call Poll Sync Rate. In an ideal world, when a Worker’s poller requests some work, the Matching service can hand it a task from its memory. This is known as “sync match”, short for synchronous matching. If the Matching service has a task in its memory too long, because it has not been able to hand out work quickly enough, the task is flushed to the persistence database. Tasks that were sent to the persistence database needed to be loaded back again later to hand to pollers (async matching). Compared with sync matching, async matching increases the load on the persistence database and is a lot less efficient. The ideal, then, is to have enough pollers to quickly consume all the tasks that land on a task queue. To have the least load on the persistence database and the highest throughput of tasks on a task queue, we should aim for both Workflow and Activity Poll Sync Rates to be 99% or higher. Improving the Poll Sync Rate will also improve the Schedule-to-Start Latency, as Workers will be able to receive the tasks more quickly.

We can measure the Poll Sync Rate in Prometheus using:

sum by (task_type) (rate(poll_success_sync[1m])) / sum by (task_type) (rate(poll_success[1m]))

In order to improve the Poll Sync Rate, we adjust the number of Worker pods and their poller configuration. In our setup we currently have only 2 Worker pods, configured to have 10 Activity pollers and 10 Workflow pollers. Let’s up that to 20 pollers of each kind.

Better, but not enough. Let’s try 100 of each type.

Much better! Activity Poll Sync Rate is still not quite sticking at 99% though, bump Activity pollers to 150 didn’t fix it either. Let’s try adding 2 more Worker pods…

Nice, consistently above 99% for Poll Sync Rate for both Workflow and Activity now. A quick check of the Matching dashboard shows that the Matching pods are well within CPU and Memory requests, so we’re looking stable. Now let’s see how we’re doing for State Transitions.

Looking good. Improving our polling efficiency has increased our State Transitions by around 150/second. One last check to see if we’re still within our persistence database CPU target of below 80%.

Yes! We’re nearly spot on, averaging around 79%. That brings us to the end of our second load, measure, scale iteration. The next step would either be to increase the database instance size and continue iterating to scale up, or if we’ve hit our desired performance target, we can instead check resource usage and reduce them where appropriate. This allows us to save some costs by potentially reducing node count.

Summary

We’ve taken a cluster configured to the development default settings and scaled it from 150 to 1,350 State Transitions/second. To achieve this, we increased the shard count from 4 to 512, increased the History pod CPU and Memory requests, and adjusted our Worker replica count and poller configuration.

We hope you’ve found this useful. We’d love to discuss it further or answer any questions you might have. Please reach out with any questions or comments on the Community Forum or Slack. My name is Rob Holland, feel free to reach out to me directly on Slack if you like—would love to hear from you.