DEV Community: Diagrid

Join the Diagrid Catalyst AWS Hackathon!

Marc Duiker — Mon, 29 Apr 2024 08:13:50 +0000

We are excited to invite you to the Diagrid Catalyst AWS Hackathon—a four-week, virtual event where you can unleash your creativity and showcase your coding skills by building or modernizing a cutting-edge application using the powerful Diagrid Catalyst platform. Get ready to collaborate, compete, and win cash prizes! Register now.

What is Diagrid Catalyst?

Diagrid Catalyst is a Developer API platform providing a brand-new approach to distributed application development. Using the Catalyst APIs, powered by the Dapr open source project, developers can overcome the complexity of rewriting common software patterns and achieve higher productivity by offloading infrastructure concerns from their code to Catalyst.

Say hello to making API calls from any platform, in any language to build modern, cloud-applications with ease. This hackathon is your chance to explore the potential of Diagrid Catalyst while bringing your solution ideas to life! For more on Catalyst, watch our most recent webinar for a platform overview and API deep dive.

Why Participate?

Join the Diagrid Catalyst Hackathon to:

Innovate with an Emerging Product: Explore the ways Diagrid Catalyst can expedite your development process by building an application to solve real-world challenges.
Collaborate with Peers: Connect with fellow developers, designers, and entrepreneurs. Collaborate, share ideas, and learn from each other's experiences.
Enhance Your Skills: Sharpen your application development and architecture skills using the coding languages of your choice and deploying your solution on Amazon Web Services.
Win Prizes: Compete for impressive cash prizes! We have rewards lined up for the most exceptional projects and for providing product feedback. ‍ All participation is evaluated based on the criteria in the Hackathon Rules, Terms & Conditions.

What to Build?

Build or update an open-source microservice-based software solution utilizing at least two Diagrid Catalyst APIs. The solution should be composed of multiple application services and either be newly written or significantly enhanced to utilize Diagrid Catalyst. The solution should be architected using AWS infrastructure and hosting services. The solution must align with one of the areas below:

Polyglot compute/ Cross-platform service discovery: Application services are deployed to different AWS hosting platforms, which interact via the Catalyst APIs.
Stateful business process orchestration: Application services are instrumented as part of a Dapr Workflow using the Dapr Workflow authoring SDK paired with the Catalyst Workflow API.
Serverless architecture: Application services are optimized to run on AWS PaaS/SaaS services, which abstracts infrastructure concerns and enables high-scalability.

Hackathon Timeline

Registration Open: April 19th 2024
Project Submissions Open: May 10th 2024
Feedback Submissions Open: May 24th 2024
Registration Closes: May 17th 2024
Project Submissions Deadline: June 7th 2024
Feedback Submissions Deadline: June 14th 2024
Location: Virtual Only

Catalyst Reference Apps

Looking for inspiration? See below for a couple of sample projects we’ve built using Diagrid Catalyst. More samples, including an AWS-specific reference, will be available to participants on the Hackathon site.

Tetris Game

The classic “blocks falling from the sky” game! The solution consists of two services:

A Python Flask user service which serves the JavaScript/HTML front-end for the user sign up and leaderboard and handles user state management and leaderboard updates.
A game service written in Go, which serves the JavaScript/HTML frontend for the game itself and manages the game session lifecycle.

These two services need to manage state and communicate to ensure games are started with the appropriate users, and user profiles are updated with the game results. The service interaction and data storage is all implemented using the APIs provided by Catalyst. Read the blog post.

Pizza Store Management System

A distributed pizza store with Catalyst APIs. The solution comprises two .NET microservices, two serverless JavaScript functions and a Vue front-end. The applications depend on Diagrid Catalyst as the serverless API layer to connect to the underlying data and messaging infrastructure. Read the blog.

How to Participate?

Participation is straightforward:

Register: Submit a registration request form for the hackathon. Eligible participants will receive a direct invite to the Hackathon site, hosted on Devpost. Accept your invitation to Devpost and click the blue “Register” button on the Hackathon landing page to register.
Create a project: Create a project to frame out your hackathon solution ideas for others to see and potentially join in to help execute. You have a few options when it comes to projects:
1. Create a closed project to participate as an individual.
2. Create an open project to enable other Hackathon participants to join you in your mission. You can even choose to invite people to participate in your project.
3. Join an existing, open project if one is available and intrigues you.
Learn: Join our ongoing Catalyst webinar series or watch on-demand.
Build: Develop your application using Diagrid Catalyst. Let your creativity shine, and don't forget to integrate innovative features.
Submit: Submit your project with the required deliverables by the deadline, June 7

For more information on resource access, project requirements and judging criteria, view the Hackathon Rules, Terms & Conditions.

Prizes

We have exciting cash prizes up for grabs! Prizes range from a Grand Prize of $2000 to Runner-up Prizes of $1000 each and Bonus Prizes from $250-$500! Get swag for participating & provide valuable product feedback for the chance to win another $100.

If you participate in the hackathon, you are also eligible for a $100 referral bonus for spreading the word! Share the Hackathon details with your friends and encourage them to sign up. The participant with the most referrals wins!

Get Ready to Hack

Don’t miss this unique opportunity to explore Diagrid Catalyst, build innovative apps, and compete for cash prizes. Whether you're a seasoned developer or a curious beginner, the Diagrid Catalyst Hackathon is your time to shine.

Building a distributed pizza store in .NET with serverless Dapr APIs

Marc Duiker — Thu, 01 Feb 2024 10:57:11 +0000

Intro

When developers build distributed systems, they need to think about cross-cutting concerns such as security, observability, and resiliency. Developers also write a lot of glue code to orchestrate services, create long-running and durable workflows, and connect to external resources, such as message brokers and state stores.

Dapr, the Distributed Application Runtime, is ideal for building distributed systems since it has built-in support for cross-cutting concerns, and it offers a suite of APIs for communication, state and workflow, that make developers more efficient. According to the State of Dapr Report, 95% of Dapr developers claim that Dapr saves time, and 55% of the developers surveyed claimed it saved them more than 30% of development time.

The usage of Dapr is also increasing thanks to a growing number of platforms that support the Dapr APIs. Dapr is no longer limited to running as a sidecar in Kubernetes, as is described in Dapr Deployment Models. In this article, Diagrid Catalyst, a suite of serverless Dapr APIs, is used to orchestrate services with Dapr workflow, and interact with message brokers and key-value stores.

Throughout this article, you’ll be using the following to configure a distributed system:

Two Dapr services (.NET)
Two serverless functions (JavaScript)
A front-end (Vue)
A key value state store
A pub/sub message broker
Diagrid Catalyst as the serverless API layer to connect everything

TLDR: Check the code on GitHub and join the Catalyst early access program.

Diagrid Pizza Store

The demo that is used in this article revolves around an online pizza store, where the user selects pizzas, orders them, and follows the progress of the order in realtime. It’s a simplified example where the durable workflow focuses on the inventory of ingredients, and communicating with the kitchen. The source code is on GitHub.

Architecture

The solution includes:

Two Dapr web services written in .NET, PizzaOrderService and KitchenService. In this demo, they’re running locally, but these can be hosted in any cloud that supports ASP.NET services.
The website (based on Vue) and two serverless functions written in JavaScript, getAblyToken and placeOrder. These are hosted on Vercel.
Ably as the realtime communication component between the PizzaOrderService and the website.
A key/value store to manage inventory (managed by Diagrid).
A pub/sub message broker to communicate between the PizzaOrderService and the KitchenService (managed by Diagrid).
Diagrid Catalyst that offers serverless APIs for communication, state, and workflow powered by Dapr.

Workflow

The PizzaOrderService contains a Dapr workflow that orchestrates the activities for checking the inventory and communicating with the KitchenService.

The workflow also contains activities for sending realtime messages to the website, these have been omitted from the workflow diagram to keep it concise.

Running the Pizza Store locally

Prerequisites

The following services, tools, and frameworks are required for this demo:

Diagrid Catalyst account (sign up for early access) and the Diagrid CLI
Vercel account (hobby) and the Vercel CLI
Ably account (free)
.NET 8 SDK
Node 18

All the code for the website, serverless functions, and back-end services is available in the catalyst-pizza-demo GitHub repository. This repository contains a devcontainer configuration that has the following preinstalled: .NET 8, Node LTS, Vercel CLI, Diagrid CLI.

You can use this devcontainer locally in VSCode (requires Docker Desktop) or directly in GitHub Codespaces. The npm install and dotnet build commands described in this article can be skipped if the devcontainer is used.

GitHub

Fork the catalyst-pizza-demo and clone it locally, or use GitHub Codespaces (recommended).

Ably

Ably is used as the serverless realtime messaging component. There is a default Ably app when you sign up for an account that can be used for this demo. Alternatively, you can create a new Ably app.

Log into the Ably portal.
Using the Ably portal: copy the Root API key from the Ably app. This will be copied later as an environment variable for both Vercel and Diagrid.

Vercel

The Vue-based front-end and two JavaScript functions are hosted on Vercel. The Vercel CLI is used to configure and run these resources locally.

Open a terminal in the root of the repository and login with the Vercel CLI:
```
vercel login
```
Go to the front-end folder and run:
```
npm install
```
Go back to the root of the repository and set up the Vercel project by running:
```
vercel
```
Follow the CLI prompts, and select the following options:
- Setup and deploy: Y
- Scope: <account name>
- Link to existing project: N
- What's your project's name? catalyst-pizza-project
- In which directory is your code located? ./front-end
- Want to modify these settings? [y/N] n
- Wait for the deployment to complete.
An environment variable is used in the getAblyToken function to generate a token for the website to communicate with the Ably realtime service. Add the Ably API token variable by running vercel env add:
- Variable name: ABLY_API_KEY
- Variable value: Use the Ably API key obtained from the Ably portal earlier
- Select Development as the environment.
Another environment variable is used in the placeOrder function to send a request to the PizzaOrderService that will start the workflow. Add the WORKFLOW_URL variable by running vercel env add:
- Variable name: WORKFLOW_URL
- Variable value: http://localhost:5064/workflow/orderReceived
- Select Development as the environment.
Run vercel pull to pull the configuration from Vercel to your local environment.
Run vercel build to build the website and the serverless functions to ensure there are no errors.

Diagrid Catalyst

Diagrid Catalyst provides serverless Dapr APIs that enables developers to quickly build distributed applications with workflow, pub/sub messaging, service invocation, and state management capabilities. Diagrid also provides managed infrastructure for storing data in a key/value store, pub/sub messaging, and workflow, which are all used in this solution.

The Diagrid CLI is used to configure the resources and run the .NET services locally.

Open another terminal in the root of the repository and use the Diagrid CLI to log in to Diagrid:
```
diagrid login
```
Create a new Catalyst project named catalyst-pizza-project and use the Diagrid managed pub/sub broker & KV store, and enable the managed workflow API:
```
diagrid project create catalyst-pizza-project --deploy-managed-pubsub --deploy-managed-kv --enable-managed-workflow --wait
```
To set this project as the default in the CLI run:
```
diagrid project use catalyst-pizza-project
```
Create a new App ID for the PizzaOrderService:
```
diagrid appid create pizzaorderservice
```
Create a new App ID for the KitchenService:
```
diagrid appid create kitchenservice
```
Before continuing, check the App ID status to make sure they are ready:
```
diagrid appid list
```

Create a pub/sub subscription for the KitchenService to receive messages from the PizzaOrderService:

diagrid subscription create pizzaorderssub --connection pubsub --topic pizza-orders --route /prepare --scopes kitchenservice

Create a pub/sub subscription for the PizzaOrderService to receive messages from the KitchenService:

diagrid subscription create preparedorderssub --connection pubsub --topic prepared-orders --route /workflow/orderPrepared --scopes pizzaorderservice

Verify that creation of the subscriptions is completed:
```
diagrid subscription list
```

Run diagrid dev scaffold to create a new local dev environment file. This creates a yaml file, named dev-.yaml with the following content:

project: catalyst-pizza-project
apps:
- appId: kitchenservice
appPort: 0
env:
    DAPR_API_TOKEN: diagrid://<dapr_api_token>
    DAPR_APP_ID: kitchenservice
    DAPR_GRPC_ENDPOINT: https://<grpc_endpoint>
    DAPR_HTTP_ENDPOINT: https://<http_endpoint>
workDir: kitchenservice
command: []
- appId: pizzaorderservice
appPort: 0
env:
    DAPR_API_TOKEN: diagrid://<dapr_api_token>
    DAPR_APP_ID: pizzaorderservice
    DAPR_GRPC_ENDPOINT: https://<grpc_endpoint>
    DAPR_HTTP_ENDPOINT: https://<http_endpoint>
workDir: pizzaorderservice
command: []
appLogDestination: ""

This file is gitignored since it contains secrets from Diagrid Catalyst. The appPort, command, and workDir attributes still need to be updated as follows:

Update the appPort for the kitchenservice to 5066.
Update the appPort for the pizzaorderservice to 5064.
Update the command arguments to ["dotnet", "run"] for both apps.
Update the workDir argument to point to back-end/KitchenService and back-end/PizzaOrderService respectively.
Update the appLogDestination to console.
Add an ABLY_API_KEY environment variable for the pizzaorderservice appId and set the value to the Ably API key obtained from the Ably portal.
Save the changes to the file.

Update the key/value store to allow shared state

The two .NET services share the same state in the key/value store to manage inventory and orders. Since this is not the default usage of state stores and services when using Dapr, an attribute needs to be set to enable this. The default behavior is that a key is prefixed with the app ID of the service that is using it. In this case, however, the keyPrefix is set to name to make sure both services use the same keys.

Run the following command to update the managed key/value store:
```
diagrid connection apply -f ./infra/kv.yml
```
This will upload the kv.yml file to Diagrid and update the configuration of the kvstore connection so keyPrefix is set to name.

Inspect the DaprClient configuration

The two .NET services use the Dapr .NET client SDK for workflow, state management and pub/sub messaging. The DaprClient is configured to use the endpoints provided by Diagrid Catalyst in the Program.cs file:

var apiToken = Environment.GetEnvironmentVariable("DAPR_API_TOKEN");
var grpcEndpoint = Environment.GetEnvironmentVariable("DAPR_GRPC_ENDPOINT");
var httpEndpoint = Environment.GetEnvironmentVariable("DAPR_HTTP_ENDPOINT");

builder.Services.AddDaprClient(options => {
    options.UseDaprApiToken(apiToken);
    options.UseGrpcEndpoint(grpcEndpoint);
    options.UseHttpEndpoint(httpEndpoint);
});

To use service invocation with the HTTP API, the HttpClient is configured with the DAPR_APP_ID and DAPR_API_TOKEN environment variables:

var appId = Environment.GetEnvironmentVariable("DAPR_APP_ID");
var apiToken = Environment.GetEnvironmentVariable("DAPR_API_TOKEN");

builder.Services.AddHttpClient(
    "daprEndpoint",
    client => {
        client.BaseAddress = new Uri(httpEndpoint);
        client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
        client.DefaultRequestHeaders.Add("dapr-app-id", appId);
        client.DefaultRequestHeaders.Add("dapr-api-token", apiToken);
    });

Running & testing the solution

Open a terminal in the root of the repository.

To restore and build the .NET projects run:

dotnet build ./back-end/PizzaOrderService
dotnet build ./back-end/KitchenService

Run diagrid dev start to start the PizzaOrderService and the KitchenService.
Using another terminal in the root of the repository, run vercel dev to start the website and the serverless functions locally.
Navigate to the URL provided by the Vercel CLI to view the website.
Select some pizzas, place an order, and watch the progress of the workflow in realtime.

Using the Catalyst API explorer

You can use the API explorer in the Catalyst web UI to interact with the APIs. If you want to retrieve the order item from the key/value store that has just been processed, follow these steps:

Open the developer tools console of the browser that is running the demo (a pizza order must be started or completed).
The order is logged to the console as a JSON object. Copy the OrderId property value.
Navigate to the Catalyst web UI. Ensure you're in the catalyst-pizza-project project.
Select App IDs in and click on the API explorer tab.
- Select the State API.
- Select the pizzaorderservice as the app ID.
- Select GET as the API operation.
- Select kvstore as the state connection.
- Enter Order- as the key, where is substituted with the value copied from the developer tools console.
Click Send. The response should show the state of the order item.

What’s next?

Congratulations! You’ve now used the Diagrid Catalyst serverless Dapr APIs for workflow, pub/sub messaging, service invocation, and state management. The ability to use these APIs from anywhere, without the overhead of managing Kubernetes clusters, brings great flexibility to developers on any platform to build distributed applications. The Dapr applications used in this demo can be hosted on any cloud. The demo uses the Diagrid managed infrastructure for key/value storage and pub/sub messaging, but these can be swapped out for other cloud-based resources, similarly to switching open-source Dapr components. You can extend this demo with an alternative pub/sub broker or state store by configuring other infrastructure connections.

Any questions or comments about this demo? Join the Dapr Discord and drop us a message in the #diagrid channel!

Dapr as the Ultimate Microservices Patterns Framework

Bilgin Ibryam — Wed, 27 Sep 2023 09:06:20 +0000

In the world of software development, microservices architecture stands out as the industry benchmark. This architectural style segments applications into distinct services, each deployable independently and organized around specific business capabilities. Such a design ensures flexibility, scalability, and resilience, with each service typically overseen by a specialized team. However, while this approach offers numerous benefits, it also introduces complexities. To navigate these challenges, developers turn to patterns, drawing parallels to time-tested design patterns from traditional software development, providing solutions for building robust distributed systems. For those seeking a comprehensive guide on these patterns, Chris Richardson’s microservices.io stands as a trusted repository, rich with insights and best practices from industry experts.

A pattern language for microservices by Chris Richardson

However, understanding patterns is just one part of the equation. Implementing them in real-world scenarios requires tools and frameworks. Positioned as a premier microservices chassis, Dapr is crafted for creating distributed applications that are secure, resilient, scalable, and observable. It doesn't merely align with the microservices patterns; it amplifies their potential, refining and simplifying their real-world implementation.

In the subsequent sections, I'll go through the patterns outlined in microservices.io, shedding light on how Dapr helps implement each. While numerous frameworks aim to address the cross-cutting concerns inherent in microservices, Dapr distinguishes itself. Its polyglot nature, sidecar operational mode, and non-restrictive stance on application architecture make it a unique and invaluable asset in the microservices toolkit.

Microservice Chassis

When embarking on the development of an application, developers often find themselves investing significant time in addressing cross-cutting concerns such as security, externalized configuration, logging, health checks, metrics, and distributed tracing. While these elements might seem straightforward, defining a curated set of dependencies that will be used in tens or hundreds of services can be a complex endeavor. The challenge amplifies when adopting a microservices architecture, given the multitude of services and the frequent creation of new ones. The Microservice Chassis pattern offers a solution by proposing the creation of a framework that serves as the foundation for microservices development. This chassis provides reusable build logic and mechanisms to handle these cross-cutting concerns, streamlining the development process.

As a polyglot framework, Dapr seamlessly handles cross-cutting concerns, allowing developers to focus on core functionalities without getting entangled in complexities. It provides built-in mechanisms for security, configuration management, logging, and more. With features like the Access Control capabilities for security, Configuration API for externalized settings, and health API endpoints for monitoring, Dapr ensures that these foundational concerns are seamlessly integrated, allowing developers to focus on core business logic without getting mired in the intricacies of these concerns. Unlike other frameworks, Dapr doesn't impose constraints on the application, granting developers the freedom to choose their preferred language, runtime, and programming style. In essence, Dapr transforms the theoretical benefits of the Microservice Chassis pattern into tangible results for real-world applications.

Sidecar

In microservices, there's often a need to augment services with additional capabilities without modifying the core service logic. The Sidecar pattern addresses this requirement. It involves deploying components of an application into separate processes or containers to provide a modular and scalable architecture. The main service runs in one container, and the sidecar service, which extends or enhances the main service, runs in a separate container but in the same network namespace. This ensures that the main service and the sidecar can communicate as if they are in the same process while being isolated from each other. The primary advantage of this pattern is the ability to separate concerns, modularize your application, and ensure that each component is focused on a specific responsibility.

Dapr is among the most popular implementations of the Sidecar pattern. When integrated into a microservices environment, Dapr runs as a sidecar alongside your service, providing a plethora of additional capabilities without requiring any changes to the main service. This includes features like state management, service-to-service invocation, pub/sub messaging, and more. By leveraging Dapr's sidecar architecture, developers can augment their services with powerful features, ensuring a robust, scalable, and feature-rich microservices ecosystem.

Service Mesh

As the number of services grows, managing inter-service communication, security, and observability becomes increasingly complex. A Service Mesh is a dedicated infrastructure layer built to handle service-to-service communication in a transparent and technology-agnostic manner. It provides features like load balancing, service discovery, observability, and security without requiring changes to the application code. By offloading these concerns to the Service Mesh, developers can focus on building business logic, while the mesh ensures that services can securely and efficiently communicate with each other.

How Dapr and service meshes compare

Dapr operates as a lightweight service mesh, providing a network layer that facilitates service discovery and ensures secure service-to-service interactions. While there are overlapping capabilities between Dapr and traditional service meshes, Dapr distinguishes itself by being developer-centric, focusing on building blocks that simplify microservices development. Unlike service meshes which are primarily infrastructure-centric and deal with network concepts like IP and DNS addresses, Dapr offers service discovery and invocation via names, a more developer-friendly approach. Beyond the common features like mTLS encryption, metric collection, and distributed tracing, Dapr introduces application-level building blocks for state management, pub/sub messaging, actors, and more. This ensures that developers gain a comprehensive toolset, not just for networking, but for holistic microservices development.

Saga

In the world of microservices, ensuring data consistency across services can be a challenge, especially when each service has its own database. The Saga pattern provides a solution to this challenge. Instead of relying on traditional distributed transactions, the Saga pattern breaks the transaction into a series of local transactions, each executed within its own service and database. These local transactions are coordinated in a specific sequence to ensure overall data consistency. If one local transaction fails, compensating transactions are executed to revert the changes made by the previous transactions. This approach offers a way to maintain data consistency without the need for distributed transactions, which are often not feasible in microservices architectures.

Dapr workflow overview

Dapr offers a concrete solution to implement the Saga pattern through its Workflow API. This API allows developers to sequence local transactions, or implement other stateful workflow patterns ensuring that data remains consistent across services. Dapr Workflow API serves as a foundational tool in this regard, streamlining the process and ensuring reliability.

Transactional Outbox

In microservices architectures, a common challenge arises when a service command needs to update aggregates in the database and simultaneously send messages or events to a message broker. The goal is to ensure atomicity - if the database transaction commits, the messages must be sent; if the database rolls back, the messages must not be sent. Traditional distributed transactions (2PC) are often not feasible or desirable due to various constraints. The Transactional Outbox pattern addresses this issue. It suggests that the service stores the message in the database as part of the transaction that updates the business entities. A separate process then retrieves and sends these messages to the message broker. This ensures that messages are sent only if the database transaction commits, preserving data consistency and order of operations.

Dapr provides a robust solution to this challenge with its Outbox feature in StateStore API. This feature allows for atomic updates to the database while also sending messages to the designated broker. By utilizing the StateStore API, developers can seamlessly integrate the Transactional Outbox pattern into their microservices, ensuring data consistency and reliable message delivery across large number of databases and message brokers.

Messaging

In the realm of microservices, reliable asynchronous communication between services is paramount. Instead of services communicating directly with synchronous calls, they exchange messages via message channels. This asynchronous mode of communication decouples services, allowing them to operate independently. It ensures that even if one service is slow or unavailable, others can continue their operations without being directly affected. This approach enhances the system's resilience, scalability, and flexibility.

Dapr's PubSub API is tailored to harness the power of asynchronous messaging for inter-service communication. By leveraging this API, developers can easily implement messaging patterns in their microservices architecture. The PubSub API ensures reliable message delivery, supports multiple messaging brokers, and abstracts the complexities of direct broker interactions.

Request/Reply Interaction

Remote Procedure Invocation (RPI) is a communication style that enables services in a microservices architecture to communicate with each other by invoking methods in a remote service. The primary advantage of RPI is its straightforwardness, allowing for direct, point-to-point communication between services. However, it's essential to manage the associated challenges, such as service discovery, reliability, encryption, to ensure the system remains resilient and secure.

Dapr addresses the challenges with its Service Invocation API. This API provides an RPI-based protocol tailored for inter-service communication in a microservices setup. By abstracting the underlying complexities, Dapr ensures that services can communicate synchronously without getting entangled in the intricacies of direct service-to-service calls. Moreover, Dapr's Service Invocation API offers built-in features like retries, error handling, and traffic control, ensuring that communications are both reliable and secure.

Circuit Breaker

In a microservices architecture, services often collaborate to handle requests. However, when one service synchronously invokes another, there's a risk that the called service might be unavailable or might exhibit high latency, rendering it essentially unusable. Such scenarios can lead to resource exhaustion in the calling service, making it unable to handle other requests. This can further cascade the failure to other services throughout the application. The Circuit Breaker pattern addresses this challenge. It functions similarly to an electrical circuit breaker. When consecutive failures cross a certain threshold, the circuit breaker "trips." For a set timeout period, all attempts to invoke the problematic service fail immediately. After this period, the circuit breaker allows a few test requests. If these succeed, normal operation resumes; if not, the timeout period restarts.

Dapr offers a concrete solution to this challenge with its Resiliency policy. This policy ensures that when the failure rate of a call exceeds a certain threshold, the call fails immediately, preventing resource exhaustion and potential cascading failures. By leveraging Dapr's resiliency policy, developers can implement the Circuit Breaker pattern efficiently, ensuring that their microservices architecture remains robust and resilient against unexpected service failures or latencies.

Access Token

In the intricate landscape of microservices, ensuring secure communication and access between services is paramount. The Access Token pattern involves issuing tokens to clients, granting them limited access to a service. These tokens encapsulate the information required to determine whether a client is authorized to perform a given operation. The primary advantage of using access tokens is that they provide a way to ensure that only authenticated and authorized clients can access services or specific operations within those services.

Dapr secure communications architecture

Dapr provides a robust mechanism to implement this pattern through its Access Control capabilities based on SPIFFE Ids. With Dapr's Access Control, developers can define and enforce policies that restrict what operations calling applications can perform on the called app. This ensures a fine-grained control over service interactions, making the system more secure and resilient against potential threats.

Service Instance per Container

The Service Instance per Container pattern deployment strategy, places each service instance in its own container. Containers, being lightweight and isolated, provide an environment where the service can run with its dependencies, ensuring consistency across different stages of deployment. This approach offers several benefits: it ensures isolation, making each service instance independent of others; it provides scalability, as new instances can be spun up quickly; and it enhances portability, as each service containts its built-time dependencies.

Dapr aligns perfectly with this deployment model as it is designed to operate best in containerized environments. When a service is deployed with Dapr, a Dapr sidecar container runs alongside the service container, enhancing its capabilities without intruding into the service's operations. Dapr's deployment model, ensures that each service instance, along with its Dapr sidecar, remains isolated in its container, benefiting from the inherent advantages of the Service Instance per Container pattern such as scalable, and resilient microservices deployments.

Service Instance per VM

In certain deployment scenarios, especially when dealing with large-scale applications or when containers might not be the optimal choice, deploying each service instance on its own Virtual Machine (VM) becomes a viable strategy. The Service Instance per VM pattern emphasizes this approach. By allocating a dedicated VM for each service instance, you ensure that the service has a dedicated set of resources, leading to predictable performance. This isolation also means that failures or resource contention in one service won't directly impact others. Moreover, VMs provide a higher degree of isolation compared to containers, which can be crucial for certain security or compliance requirements.

Dapr is versatile and can be seamlessly deployed in a VM-based environment. Whether you're deploying Dapr on its own dedicated VM or using Dapr Ambient to share its capabilities among multiple Pods, Dapr ensures that microservices can communicate and operate efficiently. This flexibility means that developers aren't confined to containerized environments and can leverage Dapr's capabilities in VM-based deployments, ensuring that the benefits of Dapr, such as state management, service invocation, and pub/sub messaging, are available regardless of the deployment strategy. Dapr's deployment documentation provides insights into how it can be integrated into various environments, including VMs, or the real edge.

Service Discovery

In microservices architectures, the dynamic nature of service instances and their locations, especially in containerized environments, presents a challenge: how does a client of a service discover the location of a service instance? Two prevalent patterns address this challenge: Client-side Discovery and Server-side Discovery. The former involves clients querying a Service Registry to discover the current locations of service instances, ensuring they always communicate with available and healthy instances. On the other hand, the Server-side Discovery pattern simplifies client code by routing requests via a knowledgeable router, often a load balancer, which interacts with the service registry.

Dapr's sidecar architecture adeptly addresses both these patterns. While the sidecar operates alongside a service, akin to a client, it isn't embedded within the application. This unique positioning allows it to query a service registry, discovering other service instances' locations, and also act as a router for inter-service calls. By offloading the intricacies of service discovery to the Dapr Service Invocation API, developers can ensure reliable service-to-service communication, even in environments where service locations change dynamically.

Service Registry

In dynamic microservices environments, the locations and number of service instances can frequently change. This poses a challenge: how can clients or routers be aware of the current available instances of a service? The Service Registry pattern offers a solution. It proposes a centralized registry where service instances register themselves upon startup and deregister upon shutdown. This registry acts as a database of services, their instances, and their locations. Clients or routers can then query this registry to discover the current locations of service instances.

Dapr offers a seamless integration with the Service Registry concept, providing a unified interface to various service registry implementations. Dapr's pluggable name resolution components used in Service Invocation API cater to diverse hosting platforms, from Kubernetes, which utilizes its DNS service, to self-hosted machines using mDNS or even HashiCorp's Consul in varied environments.

Self Registration

The Self Registration pattern ensures that services can discover each other in inter-service communication. When a service instance starts up, instead of relying on an external agent or system to register it with a service registry, the service instance itself takes the responsibility of registering. This ensures that the service registry always has the most up-to-date information about available service instances. By automating the registration process, the Self Registration pattern reduces manual intervention, potential errors, and ensures that the service registry is always current.

When a service with a Dapr sidecar is deployed, the Dapr sidecar takes the initiative to register itself with the service registry. This automated process ensures that the service is immediately discoverable by other services in the ecosystem. This not only simplifies the deployment process but also enhances the reliability and efficiency of service-to-service communication in architectures built with Dapr.

3rd Party Registration

In some microservices architectures, not all services or endpoints are created or managed by the same team or entity. There might be third-party services or endpoints that need to be integrated into the system. These third-party services might not follow the same registration patterns as internal services. With the 3rd Party Registration pattern, instead of the service registering itself (as in Self Registration), an external agent or system is responsible for registering the service with the service registry. This ensures that third-party services, which might not have the capability or permission to register themselves, are still discoverable and can be integrated seamlessly into the system.

Dapr offers flexibility in this regard too. Even non-Dapr 3rd party endpoints can be registered within the Dapr runtime, ensuring they benefit from service discovery, resiliency, and observability features that Dapr provides. This means that developers can integrate third-party services into their Dapr-enabled microservices architecture without those services being Dapr-aware. This allows a more cohesive and resilient microservices ecosystem, irrespective of the origin of the services.

Externalized Configuration

In the world of microservices, applications often interact with various infrastructure and third-party services. Examples include service registries, message brokers, databases, payment processors, and more. A significant challenge arises when trying to ensure that a service can run across multiple environments (like dev, test, staging, production) without any modifications. The Externalized Configuration pattern recommends externalizing all application configurations. This ensures that the service remains environment-agnostic and can adapt to different setups without any code changes.

Dapr secrets stores overview

Dapr provides a robust solution to this challenge with its Secrets and Configuration APIs. These APIs allow developers to externalize configurations, including sensitive information like database credentials. Instead of hardcoding configurations or placing them in easily accessible files, Dapr ensures that they are securely stored and can be fetched dynamically when needed.

Health Check API

In a microservices architecture, ensuring the health, availability, and self-healing of service instances is paramount. The Health Check API pattern proposes that each service should expose an API endpoint (e.g., HTTP /health) that indicates the health status of the service. This endpoint performs various checks, such as the status of connections to infrastructure services, the health of the host (e.g., disk space), and any application-specific logic. By periodically querying this endpoint, monitoring systems, service registries, or load balancers can determine the health of a service instance. This ensures that alerts are generated for unhealthy instances, and requests are only routed to healthy service instances, enhancing the reliability and efficiency of the system.

Dapr elevates the health check pattern by conducting periodic health checks on your service, ensuring its optimal functioning. Once app health checks are enabled, the Dapr sidecar routinely polls the application. If a health issue is detected, Dapr takes proactive measures: it unsubscribes from all pub/sub subscriptions, halts all input bindings, and short-circuits service-invocation requests, ensuring they aren't forwarded to the application. This comprehensive approach ensures that any potential issues are swiftly identified and mitigated, fostering a robust and resilient system.

Distributed Tracing

In complex microservices architectures, understanding the flow of requests across multiple services can be challenging. The Distributed Tracing pattern involves instrumenting services with code that assigns each external request a unique identifier. This identifier is then passed to all services involved in handling the request. By doing so, it becomes possible to trace the journey of a request across various services, recording information such as start time, end time, and other relevant metrics providing invaluable insights into the behavior of the system.

Dapr distributed tracing overview

Dapr automatically takes care of creating trace headers and ensures they are captured and forwarded appropriately. With Dapr’s observability capabilities, developers don't need to manually instrument their code for tracing; Dapr handles it seamlessly. Moreover, Dapr integrates with popular tracing systems, ensuring that the traces can be visualized and analyzed in a centralized manner.

Application Metrics

Application Metrics are a set of quantitative data points that provide insights into the performance, behavior, and health of an application or service. By collecting and analyzing these metrics, developers and operations teams can identify bottlenecks, detect anomalies, and optimize the performance of their services, enabling proactive monitoring and ensuring optimal system health.

Dapr automatically gathers a wide range of networking metrics, capturing data related to request rates, error rates, and latency, among others. Dapr ensures that these metrics are delivered to a centralized metrics service, allowing for comprehensive monitoring and analysis. This means that developers and operations teams can have a unified view of the system's performance, irrespective of the number of services or their complexity.

Serverless Deployment

In the evolving landscape of software development, the need for scalable, and cost-effective deployment solutions is growing. Serverless Deployment refers to a deployment infrastructure that abstracts away any concept of servers, be it physical, virtual hosts, or containers. The primary advantage is that developers can focus solely on their code, without concerning themselves with the underlying infrastructure. The serverless platform automatically scales services based on the load, ensuring optimal resource utilization.

Dapr is gearing up for the serverless era too. Soon, developers will be able to access Dapr's rich capabilities as serverless APIs. This means that the vast array of features Dapr offers, from state management to messaging, will be available in a serverless context, ensuring developers get the best of both worlds. By integrating Dapr into serverless environments, developers can ensure more robust, scalable, and feature-rich applications without the complexities of managing the Dapr infrastructure. For the updates on this pattern, follow @diagridio on Twitter and you can be among the first to try it out.

Summary

Patterns play a pivotal role in software development, serving as a shared language to communicate common challenges and best practices. They encapsulate proven solutions to recurring problems, ensuring that developers don't have to reinvent the wheel with each new project. However, while patterns provide a conceptual blueprint, they remain abstract ideas. Historically, the challenges of building reliable applications and the implementations of these patterns were addressed using application servers, service buses, and microservices frameworks like Spring Cloud, among other language-specific solutions. But to truly bring these patterns to life in the modern era, we need cloud-native frameworks and chassis like Dapr. Dapr tackles these common engineering challenges by presenting patterns as polyglot APIs, aligning with the cloud-native philosophy and offering benefits to the whole organization. This ensures that developers can leverage best practices across multiple languages and platforms, streamlining the development process and enhancing application resilience and scalability.

Top 20 Must-Read Software Trends Reports for 2023

Bilgin Ibryam — Thu, 20 Jul 2023 09:14:09 +0000

In the rapidly evolving software industry, keeping up with new trends, tools, and best practices can be time-consuming. With so much information available, where do you start, and what sources can you trust? I've curated a list of reports that I follow to stay informed and ahead of the curve. These provide insights into everything from programming languages to DevOps, cloud strategy, and security. If you're interested in the latest trends and fascinating posts I come across, follow me or check out my latest writing on industry trends over at the Diagrid blog. I share anything I find insightful and worth reading in the world of cloud and distributed systems.

Here are the top 20 reports for 2023 (in no particular order) I came across so far:

While these reports offer valuable insights, it's important to keep in mind that they can be opinionated. The key to effectively leveraging these resources lies in cross-verifying trends from multiple sources and using them only as a guide for direction rather than absolute truths.

Are there any reports that should be on this list? Tag me on Twitter and I'll include them, subject to my checks I'm always keen to explore new sources! Found this list helpful? Go ahead, share it.

Call to action: Are you a Dapr user? Your experience is valuable! Contribute your insights and shape the State of Dapr Report.

Dapr v1.11 Release Highlights

Marc Duiker — Mon, 19 Jun 2023 12:27:24 +0000

The Dapr maintainers released a new version of Dapr, the distributed application runtime, last week. This post highlights the major new features and changes for the APIs and components in release 1.11.

APIs

Dapr provides a common set of APIs for building microservices quickly and reliably. The new building block APIs, and major upgrades to existing APIs, are described in this section.

Configuration API stable

The Configuration building block allows developers to easily consume application configuration values as key/value pairs. Use this API to retrieve values that are likely to change over time but are coupled to the service. For instance, this API can be used to get App IDs, which are used when invoking other Dapr services. Or to get a component name of a state store when using the State Management API.

Applications can subscribe to changes of these configuration values, so the apps always use the latest value. The Configuration API was introduced in Dapr v1.5, and it’s great to see it now stable in v1.11.

Get a configuration value

GET http://localhost:<daprPort>/v1.0/configuration/<storeName>?key=<keyName>

Subscribe to configuration changes

GET http://localhost:<daprPort>/v1.0/configuration/<storeName>/subscribe

Service Invocation for non-Dapr endpoints 🆕

An exciting new preview feature is the ability to invoke non-Dapr endpoints. This feature enables developers to apply resiliency policies, access control via scoping, header authentication, and middleware to any endpoint. This gives you nearly all the benefits of service invocation when calling any HTTP API. Invoking non-Dapr endpoints can also be very useful when working on existing code bases where Dapr can be introduced incrementally.

Non-Dapr endpoints can be invoked in two ways:

Using an HTTPEndpoint resource type.

GET http://localhost:<daprPort>/v1.0/invoke/<HTTPEndpoint-name>/method/<my-method>

The HTTPEndpoint resource is specified in a yaml file:

apiVersion: dapr.io/v1alpha1
kind: HTTPEndpoint
metadata:
  name: <NAME>  
spec:
  version: v1alpha1
  baseUrl: <REPLACE-WITH-BASEURL> # Required. Use "http://" or "https://" prefix.
  headers: # Optional
  - name: <REPLACE-WITH-A-HEADER-NAME>
    value: <REPLACE-WITH-A-HEADER-VALUE>
  - name: <REPLACE-WITH-A-HEADER-NAME>
    secretKeyRef:
      name: <REPLACE-WITH-SECRET-NAME>
      key: <REPLACE-WITH-SECRET-KEY>
scopes: # Optional
  - <REPLACE-WITH-SCOPED-APPIDS>
auth: # Optional
  secretStore: <REPLACE-WITH-SECRETSTORE>

Using the Fully Qualified Domain Name (FQDN) of the endpoint.

GET http://localhost:<daprPort>/v1.0/invoke/<FQDN_URL>/method/<my-method>

Workflow updates

The Dapr Workflow management API has been updated to allow workflows to be paused, resumed, and purged. Purging workflows means the workflow state will be deleted. Once a workflow has been completed, it’s a good practice to purge the workflow state to keep the storage requirements to a minimum. In addition, workflows can now wait for external events. This is particularly useful for business processes that involve human interaction, such as approval workflows, or waiting for callbacks from other long-running processes.

These are the current Workflow management HTTP API methods:

POST http://localhost:<daprPort>/v1.0-alpha1/workflows/dapr/<instanceID>/start
POST http://localhost:<daprPort>/v1.0-alpha1/workflows/dapr/<instanceID>/terminate
POST http://localhost:<daprPort>/v1.0-alpha1/workflows/dapr/<instanceID>/pause
POST http://localhost:<daprPort>/v1.0-alpha1/workflows/dapr/<instanceID>/resume
POST http://localhost:<daprPort>/v1.0-alpha1/workflows/dapr/<instanceID>/purge
POST http://localhost:<daprPort>/v1.0-alpha1/workflows/dapr/<instanceID>/raiseEvent/<eventName>

Note that the Workflow management API, which is still in preview, has breaking changes compared to v1.10. The changes are outlined in this GitHub issue. Another breaking change is the ability to support multiple workflow apps in a single cluster, as mentioned in this GitHub issue.

The Dapr Python SDK has been upgraded to allow authoring of workflows, so now workflow applications can be written in either C# or Python.

Cryptography API 🆕

The Cryptography API is a new preview building block that enables developers to encrypt/decrypt data in a safe and consistent way. Cryptography can be used within key vaults or the Dapr sidecar without exposing cryptographic keys to the application. With this new API, a set of new Cryptography components is made available that is listed in the New Components section.

These examples show how data can be encrypted and decrypted with the Cryptography API using the Dapr JavaScript SDK:

Encryption

// When passing data (a buffer or string), `encrypt` returns a Buffer with the encrypted message
const ciphertext = await client.crypto.encrypt(plaintext, {
    // Name of the Dapr component (required)
    componentName: "mycryptocomponent",
    // Name of the key stored in the component (required)
    keyName: "mykey",
    // Algorithm used for wrapping the key, which must be supported by the key named above.
    // Options include: "RSA", "AES"
    keyWrapAlgorithm: "RSA",
});

Decryption

// When passing data as a buffer, `decrypt` returns a Buffer with the decrypted message
const plaintext = await client.crypto.decrypt(ciphertext, {
    // Only required option is the component name
    componentName: "mycryptocomponent",
});

For more info on how to use the Cryptography API, see the Dapr docs.

Components

Dapr decouples the functionality of the common set of APIs with their underlying implementations via components. Components of the same type are interchangeable since they implement the same interface. Release 1.11 contains both new components as improvements to existing components.

New components

Bindings

WASM: allows invoking applications compiled to WebAssembly and run in modern browsers.
Kitex: allows invoking Generic Calls in Kitex, a high-performance and strong-extensibility Golang RPC framework.

State Stores

etcd: enables state store functionality using etcd, a distributed, reliable key-value store.

Cryptography

Azure Key Vault: enables cryptography functionality using Azure Key Vault.
Local storage: enables cryptography functionality using local files. Supported file formats are PEM, JSON Web Key, and raw key data for symmetric keys.
JWKS: enables cryptography functionality using JSON Web Key Sets (JWKS).
Kubernetes Secrets: enables cryptography functionality using Kubernetes secrets.

Component improvements

With 1.11 many components have graduated to the Stable certification level, which means these can be used in production confidently. All components have a certification level as mentioned in the Certification Lifecycle. The Stable level means:

The component must have component certification tests validating functionality and resiliency.
The component is maintained by Dapr maintainers and supported by the community.
The component is well documented and tested.
The component has been available as Alpha or Beta for at least 1 minor version release of Dapr runtime prior.
A maintainer will address component security, core functionality and test issues according to the Dapr support policy and issue a patch release that includes the patched stable component.

The following components have progressed to Stable:

Bindings

AWS S3: output binding to store data in AWS S3 buckets.

Middleware

Rate limit: restricts the maximum allowed HTTP requests per second.
OpenID Connect: verifies a bearer token using OpenID Connect.

Pub/Sub

GCP Pub/Sub: enables pub/sub messaging using Google Cloud Platform Pub/Sub.

State stores

GCP Firestore: enables state store functionality using Google Cloud Platform Firestore.
SQLite: enables state store functionality using SQLite 3.

Configuration stores

PostgreSQL: enables configuration store functionality using PostgreSQL.
Redis: enables configuration store functionality using Redis.

If you're using any of these components and have a demo to show, drop a link in the #show-and-tell channel on the Dapr Discord.

*Azure components authentication improvements*

Azure AD authentication has been added to the Azure Storage Queues binding and the SQL Server state store.

All Azure components that support Azure AD now support authentication using:

Azure AD Workload Identity on Kubernetes
Azure CLI credentials.

What is next?

This post is not a complete list of features and changes released in version 1.11. Read the official Dapr release notes for more information. The release notes also contain information on how to upgrade to this latest version.

Excited about these features and want to learn more? I'll cover the new features in more detail in future posts. Until then, join the Dapr Discord to connect with thousands of Dapr users.

Creating a Dapr pluggable component for Supabase

Marc Duiker — Tue, 25 Apr 2023 14:56:41 +0000

In this post, you’ll learn how to use the Dapr pluggable component .NET SDK and the Supabase C# library to build a pluggable state store component that uses Supabase tables.

If you prefer watching a video instead of reading this blog post, head over to YouTube and watch this:

Dapr, the open-source, portable, event-driven runtime for building distributed applications, comes with over 100 built-in components which can be used to integrate resources across many clouds or on-premise systems. If those components are not enough, or you want to create something custom for your specific needs, you can create a pluggable component. The Dapr building block APIs that are pluggable are:

State management
Publish and subscribe
Input and output bindings

Since Dapr release 1.10 there are pluggable component SDKs available that make it easier to create them.

Supabase is a popular, open-source, Firebase alternative that offers Postgres databases, authentication, Edge functions, real-time messaging and file storage. The Supabase community has released a C#/.NET library that developers can use to integrate their .NET apps with the Supabase platform. For managing Postgres databases, the library uses the postgrest-csharp library that allows developers to use strongly typed models and LINQ queries.

This post describes the steps required to make a proof of concept Dapr pluggable component using .NET 7 that combines both the Dapr pluggable component .NET SDK and the Supabase C# library. The result is an ASP.NET application, that is run locally, and tested by making HTTP requests to interact with the state stored in a Supabase table.

1. Prerequisites

.NET 7 SDK
Dapr CLI
A Supabase account, you can sign up for free.
Supported OS: macOS, Linux, WSL on Windows.

2. Creating the DaprPluggableSupabase .NET project

If you’re more interested in using the pluggable component instead of creating it from scratch, continue with section 3. Supabase Setup.

Clone this repo, open a terminal and navigate to the src folder.

The cloned repository already contains an ASP.NET app in the DaprPluggableSupabase folder that is the pluggable component. You can use this as a reference.
Create a new .NET web project, and make sure to use a different name than DaprPluggableSupabase (e.g. MyDaprPluggableSupabase):

dotnet new web --name MyDaprPluggableSupabase
Change to the MyDaprPluggableSupabase folder and add the Dapr.PluggableComponents.AspNetCore and supabase-csharp NuGet packages:

cd MyDaprPluggableSupabase

dotnet add package Dapr.PluggableComponents.AspNetCore

dotnet add package supabase-csharp
Create a new class file in the MyDaprPluggableSupabase folder named KeyValue.cs.
Add the following content to this new class:

using Postgrest.Attributes;
using Postgrest.Models;

namespace DaprPluggableSupabase
{
    [Table("dapr_state_store")]
    public class KeyValue : BaseModel
    {
        [PrimaryKey("id")]
        public int Id { get; set; }

        [Column("created_at")]
        public DateTime CreatedAt { get; set; }

        [Column("key")]
        public string? Key { get; set; }

        [Column("value")]
        public string? Value { get; set; }
    }
}

The KeyValue class defines the data type that will be stored in the Supabase table. It inherits from BaseModel, which is part of the Supabase postgrest-csharp library. This library uses attributes that can be used to decorate the class and properties, which makes it easier for developers to use strongly typed models when interacting with the database table. Dapr state stores are key/value stores and Postgres databases are object-relational data stores. Since this is not a perfect match, the Dapr key/value-based state management API is mapped to this KeyValue model to fit the Postgres schema.

Create a new class file in the MyDaprPluggableSupabase folder named SupabaseStateStore.cs.
Add the following content to this new class:

using System.Text;
using Dapr.PluggableComponents.Components;
using Dapr.PluggableComponents.Components.StateStore;

namespace DaprPluggableSupabase
{
    internal sealed class SupabaseStateStore : IStateStore
    {
#nullable disable
        private Supabase.Client _supabaseClient;
#nullable enable

        private const string PROJECT_APIKEY_KEYWORD = "projectApiKey";
        private const string PROJECT_URL_KEYWORD = "projectUrl";

        public async Task DeleteAsync(StateStoreDeleteRequest request, CancellationToken cancellationToken = default)
        {
            await _supabaseClient.From<KeyValue>()
                .Where(x => x.Key == request.Key)
                .Delete(cancellationToken: cancellationToken);
        }

        public async Task<StateStoreGetResponse?> GetAsync(StateStoreGetRequest request, CancellationToken cancellationToken = default)
        {
            KeyValue? kv = await GetKV(request.Key, cancellationToken);

            if (kv != null && kv.Value != null)
            {
                var valueAsBytes = Encoding.UTF8.GetBytes(kv.Value);
                var response = new StateStoreGetResponse() { Data = valueAsBytes };
                return response;
            }

            return new StateStoreGetResponse();
        }

        public async Task InitAsync(MetadataRequest request, CancellationToken cancellationToken = default)
        {
            if (!request.Properties.TryGetValue(PROJECT_URL_KEYWORD, out string? projectUrl))
            {
                throw new InvalidOperationException($"Missing required property \"{PROJECT_URL_KEYWORD}\" in component file.");
            }

            if (!request.Properties.TryGetValue(PROJECT_APIKEY_KEYWORD, out string? projectApiKey))
            {
                throw new InvalidOperationException($"Missing required property \"{PROJECT_APIKEY_KEYWORD}\" in component file.");
            }

            _supabaseClient = new Supabase.Client(projectUrl, projectApiKey);
            await _supabaseClient.InitializeAsync();
        }

        public async Task SetAsync(StateStoreSetRequest request, CancellationToken cancellationToken = default)
        {
            var newKV = new KeyValue
            {
                Key = request.Key,
                CreatedAt = DateTime.UtcNow,
                Value = Encoding.UTF8.GetString(request.Value.Span)
            };

            KeyValue? existingKV = await GetKV(request.Key, cancellationToken);

            if (existingKV != null)
            {
                newKV.Id = existingKV.Id;
                await _supabaseClient.From<KeyValue>().Update(newKV, cancellationToken: cancellationToken);
                return;
            }
            else
            {
                await _supabaseClient.From<KeyValue>().Insert(newKV, cancellationToken: cancellationToken);
            }
        }

        private async Task<KeyValue?> GetKV(string key, CancellationToken cancellationToken)
        {
            return await _supabaseClient.From<KeyValue>()
#nullable disable
                            .Select(x => new object[] { x.Id, x.Key, x.Value })
#nullable enable
                            .Where(x => x.Key == key)
                            .Single(cancellationToken);
        }
    }
}

Notice that the class inherits from IStateStore, this is the interface that all Dapr state store components implement. The interface contains three methods that require implementing:

DeleteAsync, this will delete a value from the store.
GetAsync, this will retrieve a value from the store.
SetAsync, this will insert/update a value to the store.

All methods use the Supabase C# client to manage the data via the KeyValue model.

The IStateStore interface inherits the IPluggableComponent interface that contains only one method, InitAsync, and this method used to initialize the state store. In this case, it is used to create an instance of the Supabase.Client.

Update the existing Program.cs file with the following content:

using Dapr.PluggableComponents;
using DaprPluggableSupabase;

var builder = WebApplication.CreateBuilder(args);
var app = DaprPluggableComponentsApplication.Create();

app.RegisterService(
    "supabase",
    serviceBuilder =>
    {
        serviceBuilder.RegisterStateStore<SupabaseStateStore>();
    });

app.Run();

This class defines the application as a DaprPluggableComponentApplication and registers the SupabaseStateStore as a new state store that will communicate with Dapr applications via a Unix domain socket named supabase.

Build the application to make sure there are no compilation errors:

dotnet build

3. Supabase Setup

The pluggable component will interact with a Supabase Postgres table. This section describes how to create the table using the Supabase dashboard and where to find the project URL and API key required in the next section.

Create a new database table with the following specifications:

Name: dapr_state_store
Primary Key: id, (int8, not null)
Columns:
- created_at (timestamptz, not null)
- key (text, not null)
- value (text, null)
RLS (Row Level Security) is disabled

Note that for production use, Row Level Security (RLS) should be enabled, and access policies should be added to the table.

Table definition:

create table
        public.dapr_state_store (
            id bigint generated by default as identity not null,
            created_at timestamp with time zone not null default now(),
            key text not null,
            value text null,
            constraint dapr_state_store_pkey primary key (id),
            constraint dapr_state_store_id_key unique (id)
        ) tablespace pg_default;

You'll need the Supabase project URL and public API key to configure the Dapr component file in the next section. This information is found in the Supabase dashboard in the project Settings > API tab.

4. Update the Dapr pluggable Supabase component file

Dapr uses a modular design where functionality is delivered as a component. A component file contains the specification of a component, including the name, the component type, and related metadata that is specific to connecting with the underlying resource. The component file for the Supabase state store looks like this:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pluggable-supabase
spec:
  type: state.supabase
  version: v1
  metadata:
  - name: projectUrl
    value: ""
  - name: projectApiKey
    value: ""

The value of the spec.type field, state.supabase, consists of two parts: the component type (state), and the socket name (supabase). The socket name needs to match with the socket name argument provided in the RegisterService method in the Program.cs class of section 2. A template of this component file is available in the repository, follow these steps to update the file, so it can be used locally.

Navigate to the resources folder in this repository.
Rename the resources\pluggableSupabase.yml.template file to resources\pluggableSupabase.yml.

The pluggableSupabase.yml file is added to .gitignore, so it won't be accidentally committed to source control for this demo app. For production use, the yaml files should be checked into source control and secret store references should be used, instead of plain text values.
Open the pluggableSupabase.yml file and update the values for projectUrl and projectApiKey obtained in the previous section.
Save the file and copy it to the user's Dapr components folder: ~/.dapr/components.

When the Dapr CLI is run, all the component files in this folder will be loaded, so the pluggable Supabase component should be available.

5. Run the DaprPluggableSupabase application

Regular Dapr components are part of the Dapr runtime and don't require additional processes to run. Pluggable components, however, are not part of the Dapr runtime and need to be run separately, which is done in this section.

The DaprPluggableSupabase project in this repo implements the Dapr state store interface and uses the Supabase C# library to access a Supabase table.

Open a terminal and navigate to the
1. src/DaprPluggableSupabase folder to use the included pluggable component project or
2. src/MyDaprPluggableSupabase folder to use the pluggable component project you’ve just created yourself.
Build the project:

dotnet build
Run the project:

dotnet run

6. Run the Dapr process and test the Supabase state store

Open a new terminal and use the Dapr CLI to run the Dapr process

dapr run --app-id myapp --dapr-http-port 3500

Expected output:

The output should contain an INFO message that the pluggable-supabase component is loaded:

INFO[0000] component loaded. name: pluggable-supabase, type: state.supabase/v1

The log should end with:

ℹ️ Dapr sidecar is up and running. ✅ You're up and running! Dapr logs will appear here.
Set a new state by making a POST request to the state management endpoint:

curl --request POST --url http://localhost:3500/v1.0/state/pluggable-supabase --header 'content-type: application/json' --data '[{"key": "key1","value": "This is stored in Supabase!"}]'

Expected output:

HTTP 204 No Content
Retrieve the new state using a GET request:

curl --request GET --url http://localhost:3500/v1.0/state/pluggable-supabase/key1

Expected output:

HTTP 200 OK

"This is stored in Supabase!"

Or have a look at the Supabase dashboard to see the new state record in the table.

You've now successfully created and used the Dapr pluggable Supabase state store component. 🎉

Next steps

This proof of concept implementation of a pluggable component demonstrates how the Dapr pluggable component .NET SDK makes it easier for developers to create their custom components. You can use this solution as a starting point, and expand it with additional state management API features, such as entity tag (ETag), metadata, and options, as described in the specification. A requirement for using this pluggable component in a real project is to deploy it to a (managed) Kubernetes environment or use it in a self-hosted mode.

The Supabase platform offers a great variety of useful back-end features, and with the set of client libraries it’s very accessible to a diverse group of developers. Take a look at Supabase Launch Week to read about their latest feature announcements.

From my perspective, I’d like to explore further how Dapr can integrate with other Supabase features. It would also be great to see a Supabase state store as a built-in component that’s available in the Dapr runtime without the need of running the pluggable component separately. I also hope the proposed DocumentStore building block will get some traction this year, since this will pair up very nicely with Supabase and other PostgreSQL stores.

Do you have any questions or comments about this blog post or the code? Join the Dapr discord and post a message in the #pluggable-components channel. Have you made something with Dapr? Post a message in the #show-and-tell channel, we love to see your creations!

Create event-driven applications with Cloudflare queues and Dapr

Marc Duiker — Thu, 09 Mar 2023 14:20:59 +0000

In this post, you’ll learn how to build a cloud to edge event-driven application with Dapr and Cloudflare. You’ll learn how to create:

A Cloudflare queue.
A consumer Cloudflare worker (in TypeScript) that reads messages from the queue.
A producer Dapr app (in TypeScript) that uses the Cloudflare Queue binding to publish messages to the queue.

Event-driven applications FTW 🚀

Event-driven applications are becoming increasingly popular. The decoupling of event producers and event consumers makes applications more resilient and flexible, since they allow independent implementation and scaling of the applications.

Cloudflare recently announced Queues, allowing developers to send and receive messages with guaranteed delivery and integrating with Cloudflare Workers, a fast edge computing platform.

Dapr, the open-source distributed application runtime, is often used in event-driven applications. Dapr provides a set of standardized API building blocks that simplify microservice development. By using the Bindings building block, developers can use input, and output bindings, and either trigger their apps or invoke other resources without having to learn resource-specific SDKs for these resources. With Dapr release 1.10, a new binding is provided that allows developers to publish messages to Cloudflare Queues. Because of the common set of APIs that Dapr offers, developers from any background can use the binding to publish messages to Cloudflare Queues without needing to know the Cloudflare SDKs or adding that dependency to their codebase.

Let’s start building an event-driven app and see the binding in action.

Prerequisites

The following is required to run this sample:

Clone this repository to your local machine.
Install Dapr CLI.
- Use the Dapr CLI to install the Dapr runtime locally:
  
  dapr init
Install Node.js.
Install Cloudflare Wrangler.
Ensure you're on a Cloudflare paid plan, since that is required to use Cloudflare queues.
Enable Queues in the Cloudflare dashboard.
- Dashboard > Workers > Queues
- Enable Queues Beta
- You should see a confirmation in the dashboard that queues are enabled.

Create the applications

The solution consists of three parts:

A Cloudflare queue
A consumer Cloudflare worker that reads messages from the queue.
A producer Dapr app that will publish messages to the queue.

1. Create a Cloudflare queue

Open a terminal and use the wrangler CLI to login to Cloudflare:

wrangler login

Follow the instructions in the browser to login to Cloudflare.

The response in the terminal should end with:

Successfully logged in.
Create the Cloudflare queue using the wrangler CLI:

wrangler queues create dapr-messages

The response in the terminal should end with:

Created queue dapr-messages.

2. Create a consumer Cloudflare worker

You can either create a new consumer worker by following steps 1-3, or use the existing consumer worker in this repository and continue from step 4.

In the root folder, create a worker to consume messages:

wrangler init consumer
1. Create package.json: Y
2. Use TypeScript: Y
3. Create worker: Fetch handler
4. Write tests: N
A new folder named consumer will be created which contains the worker.

Update the consumer/src/index.ts file to:

export default {
  async queue(
      batch: MessageBatch<Error>,
      env: Env
  ): Promise<void> {
      let messages = JSON.stringify(batch.messages);
      console.log(messages);
  },
};

Add the following lines to the consumer/wrangler.toml file:

[[queues.consumers]]
queue = "dapr-messages"
max_batch_size = 1

Ensure that you're in the consumer folder and install the dependencies:

cd consumer

npm install

Publish the consumer worker:

wrangler publish

The response in the terminal should end with:

Published consumer (... sec)
  https://consumer.<SUBDOMAIN>.workers.dev
  Consumer for dapr-messages
Current Deployment ID: <DEPLOYMENT_ID>

Start a tail to read the log of the consumer worker:

wrangler tail

3. Configure the producer Dapr app

The Cloudflare Dapr binding uses a Cloudflare worker to publish messages, since only Cloudflare workers can access the queue.

There are two options for this worker:

Dapr provisions the worker.
You use a pre-provisioned Cloudflare worker.

This sample uses option 1. Read the Cloudflare Queues binding spec and choose Manually provision the Worker script if you want to go for option 2.

Create a binding file

Rename the producer/resources/binding.yaml.template to producer/resources/binding.yaml.

Open the binding.yaml file and inspect its content.

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: cloudflare-queues
spec:
  type: bindings.cloudflare.queues
  version: v1
# Increase the initTimeout if Dapr is managing the Worker for youinitTimeout: "120s"
  metadata:
# Name of the existing Cloudflare Queue (required)- name: queueName
      value: "dapr-messages"
# Name of the Worker (required)- name: workerName
      value: "dapr-message-worker"
# PEM-encoded private Ed25519 key (required)- name: key
      value: |
        -----BEGIN PRIVATE KEY-----
        MC4CAQ...
        -----END PRIVATE KEY-----
# Cloudflare account ID (required to have Dapr manage the Worker)- name: cfAccountID
      value: ""
# API token for Cloudflare (required to have Dapr manage the Worker)- name: cfAPIToken
      value: ""
# URL of the Worker (required if the Worker has been pre-created outside of Dapr)- name: workerUrl
      value: ""

The metadata.name, spec.metadata.queueName and spec.metadata.workerName values have already been set. Ensure that the queueName matches the queue setting in the consumer worker wrangler.toml file.

Values for spec.metadata.key, spec.metadata.cfAccountID, and spec.metadata.cfAPIToken still need to be provided.

Follow these instructions in the Dapr docs to set the value for spec.metadata.key.
The Cloudflare account ID should go in the spec.metadata.cfAccountID field. You can find the account ID in the Cloudflare dashboard URL: https://dash.cloudflare.com/<ACCOUNT_ID>/workers/overview.
A Cloudflare API token should go in the spec.metadata.cfAPIToken field. It can be generated as follows:
1. In the Cloudflare dashboard, go to the Workers page.
2. Click the API tokens link
3. Click the Create token button
4. Click the Use template button for Edit Cloudflare Workers
5. Update the permissions to only contain:
  - Account | Worker Scripts | Edit
6. Update the Account Resources to only contain:
  - Include | <YOUR ACCOUNT>
7. Set a time to live (TTL) for the token, the shorter, the better, if you're just testing.

Now the binding file is complete. The file is gitignored, so the secrets won't be committed to the repository.

Inspect the Node app

Let's have a look at the Dapr app that will send messages to the Cloudflare queue.

Inspect the producer/index.ts file.

import { DaprClient } from "@dapr/dapr";

// Common settingsconst daprHost = "http://localhost";
const daprPort = process.env.DAPR_HTTP_PORT || "3500";

async function main() {
    console.log("Starting...");

    const bindingName = "cloudflare-queues";
    const bindingOperation = "publish";
    const client = new DaprClient(daprHost, daprPort);
    for(var i = 1; i <= 10; i++) {
        const message =  { data: "Hello World " + i };
        const response = await client.binding.send(bindingName, bindingOperation, message);
        if (response)
        {
            console.log(response);
        }
        await sleep(1000);
    }

    console.log("Completed.");
}

async function sleep(ms: number) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }

main().catch((e) => {
    console.error(e);
    process.exit(1);
})

Note that the bindingName is set to cloudflare-queues and matches the value of the metadata.name in the binding.yaml. The bindingOperation is set to publish (create could be used as an alias).

Run the producer app

Open a new terminal window and navigate to the producer folder.
Install the dependencies:
```
npm install
```

Run the following command to start the producer app:

dapr run --app-id producer --resources-path ./resources -- npm run start

The terminal that logs the tail of the consumer app should show a log statement for each of the ten messages sent:

Unknown Event - Ok @ 17/02/2023, 11:22:50
  (log) [{"body":"{\"data\":\"Hello World 1\"}","timestamp":"2023-02-17T10:22:50.556Z","id":"8f6293d9d04001e3f2a12be5c47acde2"}]
...

Congratulations! You’ve now sent messages from a Dapr app to a Cloudflare Queue (& Worker) via the Cloudflare Queues binding! 🎉 Hungry for more Dapr bindings? Have a look at the long list of supported bindings.

Any questions or comments about this blog post or the code? Join the Dapr discord and post a message in the #components-contrib channel. Have you made something with Cloudflare and Dapr? Post a message in the #show-and-tell channel, we love to see your creations!

Dapr 1.10 release highlights

Marc Duiker — Tue, 21 Feb 2023 08:23:41 +0000

The Dapr maintainers released a new version of Dapr, the distributed application runtime 🎉. This post highlights the major new features and changes in release 1.10, the first one in 2023.

Workflows

My favorite new feature is the workflow engine and building block. This feature enables developers to orchestrate business logic for messaging and state management across various microservices. Workflows are suitable for multistep processes such as order processing, HR onboarding, and coordinating software updates across many devices because workflows are stateful and support long-running services. Workflow patterns such as chaining and fan-out/fan-in are written in code to execute tasks reliably. The built-in workflow engine is durable and resilient and can either be used via HTTP/gRPC, as well as the .NET SDK (other languages will follow).

Read more about the Workflow building block in the Dapr docs or try out the Workflow Quickstart.

Pluggable Components SDKs

Dapr provides a long list of built-in components to work with. But if this isn’t enough, Pluggable Components allows developers to create their own components for their specific integration requirements. Pluggable Components can be written in any language that supports the gRPC protocol.

With release 1.10, developers can create components using the Pluggable Components SDKs in .NET, Java, and Go instead of using the gRPC protocol directly. This will enable many developers to create new components, both public and private ones.

Read more about Pluggable Components in the Dapr docs.

Running multiple Dapr apps

Running a microservices system locally can be challenging, since multiple apps need to be up and running. Developers would need to call the dapr run command for each microservice separately, which gets a bit tedious.

With release 1.10, another big developer productivity improvement has been achieved. Developers can now run multiple apps with just one command and a multi-app template file. The dapr run -f <path> command is used to run Dapr locally, where points to a directory that contains a dapr.yaml file, the multi-app template, that contains the properties to run the apps. More information on how to structure the multi-app template can be found in the Dapr docs.

Components

Dapr uses a modular design where functionality is delivered as a component. Release 1.10 contains both new components as improvements to existing components.

New Components

SQLite: Enables state store functionality using SQL Lite 3. SQLite is a small, fast and self-contained SQL database engine, great for local development and running on small devices.
Cloudflare KV: Enables state store functionality using Cloudflare Workers KV. Workers KV is a global, low-latency, key-value data store that is ideal for high read volumes.
Cloudflare Queues binding: Enables publishing message to Cloudflare queues. Cloudflare queues allow developers to send and receive messages with guaranteed delivery. It integrates with Cloudflare Workers and offers at-least once delivery and message batching.
KubeMQ binding: Supports input and output bindings for KubeMQ, an enterprise-grade Kubernetes-based message broker and message queue that is scalable, highly available, and secure
KubeMQ: Enables pub/sub messaging using KubeMQ.
Solace AMQP: Enables pub/sub messaging using Solace PubSub+ Event Broker. The Solace Event Broker is a container-based message broker that enables real-time data distribution between applications.

Component Improvements

With 1.10 many components have graduated to the Stable certification level. All components have a certification level as mentioned in the Certification Lifecycle. The Stable level means:

The component must have component certification tests validating functionality and resiliency.
The component is maintained by Dapr maintainers and supported by the community.
The component is well documented and tested.
The component has been available as Alpha or Beta for at least 1 minor version release of Dapr runtime prior.
A maintainer will address component security, core functionality and test issues according to the Dapr support policy and issue a patch release that includes the patched stable component.

The following components have progressed to Stable:

AWS SQS/SNS: Enables pub/sub messaging using AWS SQS or SNS.
AWS DynamoDB: Enables state store functionality using AWS DynamoDB.
CockroachDB: Enables state store functionality using CockroachDB.
MySQL: Enables state store functionality using MySQL.
HashiCorp Vault: Enables secret store functionality using HashiCorp Vault.
Pulsar: Enables pub/sub messaging using Apache Pulsar.
Azure Service Bus Queues binding: supports input and output binding to Azure Service Bus Queues.
Cron binding: Enables scheduled task execution via cron expressions.

If you’re using any of these components and have a demo to show, drop a link in the #show-and-tell channel on the Dapr Discord.

What is next?

This post is not a complete list of features and changes released in version 1.10. Read the official Dapr release notes for more information. The release notes also contain information on how to upgrade to this latest version.

Excited about these features and want to learn more? I’ll cover the new features in more detail in future posts. Until then, join the Dapr Discord to connect with thousands of Dapr users.