DEV Community: Relay

Announcing Relay's General Availability Launch

Melissa Sussmann — Tue, 06 Apr 2021 00:00:00 +0000

Today we’re proud to announce the general availability of Relay, a cloud-native workflow automation platform. We launched our public beta of Relay last June, and we’re now officially out of beta and open for business! We’ve been pretty busy during the beta period - early users have executed thousands of workflows, processed tons of events, and given us incredibly helpful feedback.

We believe there is tremendous demand for a new kind of low-code, responsive automation product because:

The cloud has radically changed how we build and operate systems.
Lower-level infrastructure components are, for the most part, good enough at the problems they are intended to solve.
Complexity has moved up the stack, beyond configuring operating systems and into how we tie services, APIs, and distributed systems together.
DevOps teams encompass a wide range of skills, and it’s important to the business that their tiny number of automation specialists democratize their knowledge across the organization.

Thus, how we automate across our infrastructure must change accordingly.

We’ve heard over and over from our users how their quotidian tasks are deceptively tricky and involve sequencing lots of actions across all manner of different services. Going through these tasks manually introduces room for error, even when they’re properly documented. Between responding to service-down incidents, rolling back failed deployments, and securing cloud resources… the struggle is real.

Solving these problems involves gluing together a patchwork of existing scripts, bespoke in-house APIs, and 3rd-party services to get anything done. This emerging need for better orchestration is why Relay is built around event-driven workflows. What does that mean, exactly?

Modern service architectures generate all kinds of events — mostly noise, but with important signals intermixed — and the ability to understand and respond to those events automatically is key. YAML-based workflows provide a readable, reusable abstraction that the whole team can comprehend and iterate on. They’re well-suited for assembling individual automation “steps” into an end-to-end solution. So, combining workflows and events leads to truly responsive automation that can cover the full continuum of scenarios ops folks are regularly faced with, at the velocity they need.

Powerful abstractions like events and workflows are great, but not if they come at the expense of accessibility or if they present users with more of a learning cliff than a gentle curve. One thing we learned during the beta was that users wanted the best of both worlds: a simple workflow authoring experience that doesn’t require much coding, but that is also harmonious with their overall infrastructure-as-code approach. This is why Relay takes a low-code approach to workflow authoring. We’ve spent a lot of time making that experience quick and easy, but all the while changes are bi-directionally synced to human-friendly code.

It’s been wonderful seeing the automation problems that our users have solved with Relay. The major themes that have arisen are:

Self-healing infrastructure - using Relay to add intelligence to existing automation tools, driving them in response to high-signal events, and integrating them into higher-level, auto-remediation workflows, e.g. combining Relay with Puppet Enterprise or doing automated rollback of a complex deployment.
Compliance - using Relay to continually verify the security posture of key cloud resources by receiving infrastructure events and then applying the right compliance policies. This is such an important issue for users that we plan to do a lot more on this front. Stay tuned for more on this soon!
Incident response - using Relay to enrich alert data, automate incident communications, and trigger auto-remediation workflows, e.g. our partnerships with PagerDuty, DataDog, and Splunk.

Case Study

One of our current customers, Bryxx NV, is a Belgium-based managed service provider that is consolidating its cloud automation stack onto Relay. Bryxx manages its customers’ multi-cloud infrastructure and collects telemetry in Grafana. We worked together to build a Grafana integration for Relay with two parts: Relay receives threshold alerts from Grafana when additional capacity is needed. After running workflows that handle the scale-up, Relay posts an annotation back into Grafana, so there’s a record of the workflow run overlaid on the dashboard, providing an audit trail and visual record of the changes.

Before using Relay, Bryxx had parts of these operations automated but still had to manually coordinate and orchestrate the changes. Now, Bryxx’s DevOps Architect Dries Dams says, “Relay helps us connect all the dots, achieving true self-healing systems on cloud-native platforms. The ease with which we can create new workflows saves us countless hours of developing custom scripts leaving more time for our engineers to help our customers grow their business.”

Next steps

The next step, and the best step, is to try it out (for free)! There are two additional tiers that complement the free Community tier:

Team : ($20 user/month) For small to mid-sized teams, this tier provides access to up to 30 users, 500 active workflows, Role-Based Access Control (RBAC), and Single Sign-On (SSO).
Enterprise : For large organizations with on-prem needs, this tier provides up to 5,000 active workflows and up to 5,000 users. It also provides RBAC and SSO and on-prem connectivity with Puppet Enterprise, Puppet’s flagship product. Contact sales for pricing.

If you’d like to learn more about Relay:

How to get involved to extend Relay to better meet your needs and become part of the Relay community.
The documentation site introduces Relay, its usage, core concepts, and extension points.
Join the Puppet community Slack - come linger in the #relay channel! The more the merrier.

Thanks, and happy automating!

How to Remediate Unencrypted S3 buckets

Relay — Tue, 26 Jan 2021 19:29:05 +0000

Introduction

Cloud environments are always susceptible to security issues. A significant contributor to this problem is misconfigured resources.

Traditional IT Infrastructure was somewhat static; server hardware only changed every few years. With few changes occurring, security was also more static. The modern cloud environment is a much different challenge. In cloud environments, servers, services, and storage are created with automation, resulting in a dynamic and potentially ever-changing server environment.

Standardized policies and regular enforcement of best practices are key to reducing security risks. New automation can be created to enforce these policies on an ongoing basis. Even if the configuration drifts, automation can pull systems back into compliance.

Unencrypted S3 buckets are an example of a configuration setting that could expose enormous quantities of sensitive data. In this article, we will look at using Relay to enforce encryption that is easy to set up and monitor.

Setting Up Relay for AWS

Relay is an online platform with Graphical User Interface (GUI) and Command Line Interface (CLI) for all your cloud automation use cases. In our example, we will be using the GUI web interface.

You will need to set up a Relay account at https://app.relay.sh/signup to create and run a workflow. The workflow will force encryption on specified S3 buckets in your Amazon Web Services (AWS) account.

The Relay AWS connection requires creating an Identity and Access Management (IAM) user with permissions to edit S3 buckets. In your AWS console, go to the IAM dashboard and set up an IAM user. For example, under “Access management,” I set up a group called “BucketGroup” and a user named “relaydemouser”. Make sure you keep a copy of the “Access key ID” and “Secret access key” as you will need these. The “Secret access key” is only shown once.

The user/group will need permissions to access the S3 buckets. For the example above, I added the policy “AmazonS3FullAccess” to the permissions of the BucketGroup. Do the same with the user/group you create.

This policy gives the IAM user, or group of users, full access to the S3 buckets, including checking configuration, listing buckets, and encrypting S3 buckets.

Once you have a user with the required permissions, you can set up the Relay connection configuration. Open the Connections section of Relay and add a connection to AWS with this IAM user.

You can choose any name for the connection but will need to use the AWS “Access key ID” and “Secret access key” for your IAM user as shown below:

Creating a Workflow

Relay has many sample workflows, including one to remediate unencrypted S3 buckets. Use this link to find it, or, from inside the GUI, click on the workflow icon and then Explore workflows. Under the Security heading, you will see “Remediate unencrypted S3 buckets.”

Looking at a new AWS S3 Storage bucket’s properties page, you will see that encryption is disabled by default.

The workflow will enable server-side encryption. After running the workflow, your default encryption settings will look like this:

You can review the workflow contents (code, graph, and so on) then click on the “Use this workflow” button.

Clicking Try this workflow will bring you to a new workflow dialog with a suggested name. Note that workflow names must be unique within your account. Click on Create workflow and Relay will display the workflow graph:

To run the workflow, click Run in the top right corner of the workflow page. Running the workflow will bring up a dialog showing dryRun = true. Doing a dry run will test the logic but will not make any changes. Go ahead and run the workflow in dryRun mode. Visit the Using Workflows section of the documentation for more information on workflows.

The workflow will stop at the approval step and will not execute the action step to encrypt the S3 bucket. In dryRun mode, the Yes and No approval buttons are disabled. If you rerun the workflow and set the dryRun dialog to false, Relay will stop the workflow at the approval step and allow you to click on “Yes” or “No.” If you click on “Yes,” the flow will continue and encrypt the S3 buckets.

After the workflow is complete, go back to AWS and check the encryption status on your S3 bucket. The buckets that had previously been unencrypted should now be encrypted.

Setting up a Trigger

Relay supports multiple ways to trigger a workflow. We have already walked through the manual trigger, running a workflow manually. You can also schedule a trigger to run automatically, using a process similar to that of a Linux cron job. Relay also supports webhook and REST API triggers. Services which support webhooks can post a JSON payload to Relay when an event happens. The Relay REST API receives JWT-authenticated requests from a remote system and runs a workflow in response. Read all about triggers in the Using Triggers section of the documentation.

For this example, we’ll use a simple schedule trigger to run the workflow at midnight every day. Click on the “Add trigger” button in the first block of the graph to display the Add trigger dialog.

Clicking on “Run a trigger every day” will bring up a code snippet that will be added to your workflow.

Once you save the modified code, you will see that the trigger block in the graph now says it will run every day. A summary of the five digits of Cron time setting ‘* * * * *’ is shown below—an asterisk (*) means no specific value.

Position	Time Unit	Possible Values
1	Minute	0 to 59 or *
2	Hour	0 to 23 or *
3	Day of Month	1 to 31 or *
4	Month	1 to 12 or *
5	Day of Week	0 to 7 or * Both 0 and 7 represent Sunday.

When you run the workflow, it will run immediately, plus run automatically every day. You would need to remove or modify the approval step if you want the workflow to run continuously.

You can remove the schedule trigger by deleting the triggers section in the code and saving changes. The graph button will change back to “Add trigger.”

Next Steps

You have configured a workflow to add encryption to AWS S3 storage buckets, and learned to set up a scheduled trigger. As described in this article, many preconfigured workflows make it easy to get started with Relay. Plus, the code is readily available for modification and expansion outside the GUI interface.

Other example Relay flows are available for many Azure, AWS, and GCP tasks.

Check out Relay and start automating your DevOps maintenance today.

Save Time and Money by Automatically Deleting Unused Azure Load Balancers

Relay — Tue, 12 Jan 2021 00:00:00 +0000

Using the cloud reduces on-premises infrastructure costs and related maintenance. Instead of deploying more servers, storage, and networking components to your own datacenter, you are now deploying these as cloud resources. Using the cloud is supposed to reduce infrastructure and maintenance costs. However, deploying cloud resources also risks over-commissioning, under-usage, and keeping resources running that are not always needed or, even worse, no longer in use.

To help you avoid wasting these unused resources, Puppet created Relay. This tool enables you to automate DevOps cloud maintenance, including automatically cleaning up resources you no longer need. This reduces resource waste while saving DevOps time, helping your team focus on delivering exciting new product features.

Cleaning up Azure

In this article, we walk you through a common scenario. You may be using Azure Infrastructure components, like Virtual Machines and related Virtual Networking resources, together with Azure Load Balancers. When you no longer need the Virtual Machine, and delete it, you may forget about the Azure Load Balancer. The Load Balancer then continues to use your valuable cloud computing resources. Relay workflow helps you clean up.

To use Relay:

Activate your Relay account at relay.sh.
Define a connection between Relay and Azure, using an Azure service principal.
Create your Relay workflow, which then performs the cleanup.

Let’s examine each of these steps with step-by-step guidance. If you already have an Azure subscription with administrative access, you can follow these steps in your own environment.

Setting Up Relay

Setting up a Relay account is rather straightforward. It is a hosted-cloud service with nothing to download, install, update, or maintain.

First, create a new Relay account by completing the required fields on the signup page.

After accepting the activation link in your email, create a complex password, confirm it, and that’s all it takes.

Then, after successfully logging on to the Relay platform, you are ready to start. From the Relay portal, browse sample workflows or create a new one from scratch.

There is also an option to define connections, where you specify the service account of your cloud platform. Relay supports many different public and private cloud environments, including Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), and Kubernetes. In our example, we will use Azure, but the process is similar in all environments.

Configuring Access to Azure

Before creating a workflow, let’s start by defining a connection for Azure. This relies on an Azure Active Directory service principal. Think of this object as just another user account, used by an application and similar to an Azure administrative account.

Once you create the service principal, apply Azure Role Based Access Control (RBAC) permissions, limiting this account’s administrative capabilities to keep your production environment secure. Optionally, you can read more about service principal objects.

There are several ways to create a service principal, such as Azure Portal, Azure Cloud Shell, PowerShell, ARM Template, or REST API. We will show you how to create the service principal using Azure Cloud Shell.

First, navigate to the Azure Portal, and open Azure Cloud Shell from the top right menu. Note: if this is the first time you use Azure Cloud Shell, it will ask you to create an Azure Storage Account and Azure FileShare – complete this step to continue. Select Bash as the interface.

Type the following Azure command-line interface (CLI) command:

az ad sp create-for-rbac -n "Relaysp"

This creates the service principal in your Azure Active Directory, and shows the actual account credential details as Shell output. Copy this information for later use.

Go to the Relay portal and select “Connections”. Next, click the “Add Connection” button and choose Azure from the list. This opens the “Set up your Azure Connection” window. Complete the fields, copying the information from the Cloud Shell output, as shown in the below example:

Relay Names Azure Service Connection Names

Subscription ID - Subscription
Client ID - appId
Tenant ID - tenant
Secret - password

Save the information. This creates your connection.

We are now ready to create our workflow.

Creating a Workflow

There are a couple of different ways to create the workflow. Remember, it is a YAML dialect, allowing us to author the file in any intelligent text editor, like Visual Studio Code or similar.

You could write the YAML from scratch, but Relay provides an extensive open source library of sample workflows on GitHub, integrated into the Relay portal. So, let’s have a look.

First, from the Relay portal, navigate to Workflows. Select “Explore our workflow library_”._

From the category list, select “Cost optimization”.

Locate the “Delete empty Azure Load Balancers” workflow.

Click “Use this workflow”, provide a unique name, and confirm by pressing “Create workflow”.

This imports the workflow into your dashboard. From here, you need to complete some minimal configuration settings.

The first warning we get is “missing required connection”. You need to specify which Azure Connection the workflow should use, namely, the one you created in the previous step. To fix this error, click the “Fill in missing connections” link.

Here, it shows a default Connection of “my-azure-account”. You could add a new connection here, however, we already created one. To use it, we need to switch back to the Code view of the workflow and make a change in the YAML.

In the code editor, search for the keyword “connection” (line 27 in the sample file), and replace the name “my-azure-account” with the name of the Azure Connection you created earlier (Azure-Relay in our setup). Make sure you save the changes.

Notice how the connection is ‘recognized’ under the settings pane to the right.

From here, let’s test the workflow to ensure it can successfully connect to our Azure subscription and detect Load Balancers. Click “Run” and confirm by pressing “run workflow” in the popup window. Also, notice this flow can run in dryRun mode, which means it won’t actually change anything in our environment.

The workflow kicks off and displays the step-by-step sequence:

The dry run workflow worked fine. To look at each step in a bit more detail, hover over the step, for example, “list-azure-load-balancers”, and select view logs.

The following output is displayed:

As you can see, the workflow lists the valid load balancers. Note: we didn’t deploy a Load Balancer yet, so it is normal that the result is zero, but at least it ran fine.

Remember, this workflow is checking for “empty load balancers”, which means it looks for Azure Load Balancers without any endpoint connection parameters. To make this a more viable test, let’s deploy a Load Balancer in Azure. If you need some assistance on how to do this, you might use this sample from Azure QuickStart Templates.

This deploys a Standard-type Azure Load Balancer, together with three Virtual Machines (VMs) as a back-end pool. If you want, you can update the Azure deployment template to deploy only a single VM if you don’t want to test three.

After deploying this template, the setup looks similar to this:

Let’s go back to Relay and run our workflow once more. Notice that, this time, the Load Balancer is actually detected.

Next, select the next step in the flow, filter-loadbalancers, and view logs. This reveals that the detected Load Balancer is not empty.

It is interesting to know how Relay identifies if the Load Balancer configuration is empty or not. Let’s have a look at the actual YAML code.

The script includes a pointer to a Python script, which looks like this:

You can see it is checking for any Load Balancers with empty “backend_address_pools”.

To run a more valid test, let’s go back to the Azure Load Balancer, and clear the backend pool configuration. The easiest way to remove this is by using the following PowerShell cmdlet:

Remove-AzureRMLoadBalancerBackendAddressPoolConfig

For more information on doing this, refer to the following Microsoft Doc:

Remove-AzureRmLoadBalancerBackendAddressPoolConfig

Note: if you used the default settings from the Azure Quickstart Template sample deployment earlier, use the following PowerShell script:

Get-AzureRmLoadBalancer -Name "MyLB-lb" -ResourceGroupName "MyLBRG" | Remove-AzureRmLoadBalancerBackendAddressPool -Name "LoadBalancerBackendPool" | Set-AzureRmLoadBalancer

The Load Balancer configuration shows an empty BackEndPool now:

Run the Relay workflow once more to see what happens with the Azure Resource. To test, make sure you keep the DryRun value set as True. The result is the following:

This time, the empty Load Balancer is detected for removal.

Next, trigger the workflow once more, this time setting the DryRun to False, which means it will actually force the removal of the Azure Load Balancer resource. You will also need to confirm “Yes” on the approval step in the workflow.

After about a minute, we get positive confirmation saying all specified load balancers are deleted. Validating from the Azure Portal confirms this.

As a last step in this process, I want to highlight the trigger option at the beginning of the workflow. Up till now, we kicked off the workflow manually. This works fine for testing, but doesn’t work well in production. You may want to schedule this cleanup, validating your environment every week or maybe every night.

To schedule your load balancer cleanup, first, select the first step in the workflow, “Add trigger”.

This opens a list with different triggers to choose from, like running a “per day” job or triggering the workflow using an HTTP interaction. This could be interesting if you want to automate the process from your systems management tool. Whenever you connect to the trigger-HTTP-URL, the workflow executes. Let’s go back to the day-trigger scenario for now.

Select the “Run a trigger every day” option, which reflects the corresponding code. The most important setting here is “schedule”, using the standardized cron notation to define the scheduling format.

For more information on the cron definition, I can recommend this helpful link. You now have one less thing to monitor. You can check it off your busy DevOps team’s to-do list.

Next Steps

In this article, we introduced you to Relay, a product Puppet created to automate cloud management tasks. From first configuring the cloud connection, we explored an example scenario, cleaning up empty Azure Load Balancers. We then learned how to trigger the workflow as a daily scheduled task.

Other example workflows enable you to easily set up maintenance of VMs, network interface controllers (NICs), and Disks.

While this example detailed how to clean up Azure, Relay also works with other cloud environments. You can adjust the example above to develop workflows for any Azure, AWS or GCP resources.

Automating your load balancer cleanup and other DevOps tasks saves you time and money as your team instead focuses on creating great new features for your software applications. Check out Relay to automate your DevOps maintenance today.

Deployment Rollbacks via FireHydrant Runbook

🌈 eric sorenson 🎹🔈🎚 — Tue, 24 Nov 2020 00:00:00 +0000

FireHydrant has a sophisticated set of response actions for coordinating communications, activities, and retrospectives for incidents that affect your services. Relay helps by automating remediations that involve orchestrating actions across your infrastructure. In this example workflow, an incident that affects an application deployed on Kubernetes can trigger a rollback to a previous version automatically.

The workflow makes a couple of assumptions about your infrastructure that will need to be true to work out of the box; we’d love to work with you if you need additional flexibility! Specifically, it maps FireHydrant “Services” to Kubernetes deployments, and FireHydrant Environments to Kubernetes namespaces. Your deployment and rollback process is likely different from the one modelled here, but this should provide a good starting point for automating incident response activity with Relay.

Connecting the Services

In order to enable two-way communication between FireHydrant and Relay, we’ll need to do some work on both sides: Relay posting to FireHydrant requires a FireHydrant API key, and FireHydrant triggering Relay workflows uses a dynamically-generated Relay webhook URL.

First, add the Relay workflow to your account using this link. When you click “Save”, Relay will both create the webhook URL you’ll need and prompt that you’re missing a Secret and a Connection - we’ll get to those in a moment.

In FireHydrant, we’ll create a Runbook that will trigger the workflow by sending a webhook to Relay. Create a new Runbook and add a Send a Webhook step. For the Endpoint URL , paste the webhook address from the Relay’s Settings sidebar. The HMAC Secret field is an arbitrary string (not currently used). For the JSON Payload field, paste the following template:

{
  "incident_id": "{{ incident.id }}",
  "name": "{{ incident.name }}",
  "summary": "{{ incident.summary }}",
  "service": "{{ incident.services[0].name | downcase }}",
  "environment": "{{ incident.environments[0].name | downcase }}",
  "channel_id": "{{ incident.channel_id }}",
  "channel_name": "{{ incident.channel_name }}"
}

Next, create a FireHydrant API key for Relay to post information back into the incident timeline. Under Integrations - Bot users in FireHydrant, create a new Bot user with a memorable name and description. Save the resulting API token into a Relay secret on the Relay workflow’s Settings sidebar named apiKey (case-sensitive).

GCP Authentication Setup

This workflow uses a GCP Connection type on Relay’s end, which requires a service accountconfigured on your cluster. Follow the GCP guide to API Server Authentication’s “Service in other environments” section to set one up. This workflow will require the service account have the role roles/container.developer attached to it; if you re-use the connection for other workflows it may require additional permissions. Once you’ve gotten the service account JSON file downloaded, add a GCP Connection in Relay, name it relay-service-account and paste the contents of the JSON file into the dialog. Under the hood, Relay stores this securely in our Vault service and makes the contents available to workflow containers through the !Connection custom type in the workflow.

For non-GCP clusters, you can use Relay’s Kubernetes connection type, which requires less setup. GCP rotates access tokens every hour, making them unsuitable for automated use. The Kubernetes connection type needs an access token, the cluster URL, and the CA certificate for the cluster; there are more detailed instructions accompanying this deployment workflow example.

Configuring Services and Environments

One of the FireHydrant’s big benefits is its awareness of your infrastructure. It takes a little bit of up-front work, but if you invest the time to map out your services and environments, you can dramatically streamline your incident response. In this example, we’ve enumerated the microservices that make up our Sock Shop application and associated them with different runbooks. Fortunately for the demo, the sockshop-frontend service is a simple stateless Deployment in GKE, which makes new releases easy to manage with the kubectl rollout command.

Similarly, the Environments section lets you enumerate the instances of your service, to better characterize the impact of an incident, help assign owners for remediation actions, and message outage information to the appropriate audiences. Check out this FireHydrant helpdesk article on inventory management for more details on infrastructure organization. For our purposes, the goal of defining environments is to map them onto Kubernetes namespaces where our application is running. (For production workloads, it’s more likely that your environments map to distinct clusters; that’s totally possible to handle in Relay but is beyond the scope of this introduction!)

Incident Creation and Response

Now for the exciting part. Let’s say an update bumped the image on the frontend pods from a pinned version to latest and everything broke.

% kubectl set image deployment/sockshop-frontend nginx-1=nginx:latest \
   --record --namespace production deployment.apps/sockshop-frontend
image updated
% kubectl rollout history deployment sockshop-frontend --namespace production
deployment.apps/sockshop-frontend
REVISION CHANGE-CAUSE
1 kubectl set image deployment/sockshop-frontend nginx:1.18.0=nginx:latest --record=true --namespace=production
2 kubectl set image deployment/sockshop-frontend nginx-1=nginx:latest --record=true --namespace=production

But unbeknownst to the deployer, all was not well. After some troubleshooting, we determined that the application was degraded and rollout was to blame. In FireHydrant, the on-call person declares an incident and indicates the service and environment that were affected.

Once the incident is created, we can attach the rollback-via-relay Runbook that contains our webhook to the incident. The benefit of doing it this way is that the credentials are stored in Relay rather than needing a comamnd-line kubectl setup as above, and you don’t have to remember the exact syntax to type if something’s broken at 4AM! The correct steps are stored in the workflow, reducing the possibility for an error.

The incident timeline and associated Slack channel show these action taking place, and in Relay we can see the webhook come in, the workflow kick off, and ultimately post back into the timeline with the output of the rollback command. Thanks to FireHydrant’s awesome Slack integration, the updates roll into the channel in real time and chat messages are mirrored back into the incident so teams can coordinate their activities and keep a record of what happened. In this case, the rollback worked and we can resolve the issue quickly!

Conclusion and Next Steps

FireHydrant’s Runbook based system for coordinating actions in response to incidents is extremely powerful. The more you teach it about your infrastructure, the faster you’ll be able to respond when something goes wrong. And linking it to Relay via Runbooks enables another level of automated response and remediation. In this example, we were able to roll back a deployment without someone needing to run manual commands and potentially making things worse!

There are lots of existing Relay workflows that can act as building blocks or examples to construct your own incident response workflow. By combining them with clear processes codified in FireHydrant, responders can solve issues more quickly, reduce downtime, and get back to higher-value work.

To try this out for yourself, sign up for free FireHydrant account and Relay accounts, and get started automating!

Happy Hacktoberfest

Melissa Sussmann — Tue, 27 Oct 2020 00:00:00 +0000

Happy Halloween! ‘Tis the spooky season and this year, we are kicking off a Hacktoberfest challenge just in time for Halloween. All you have to do for this Hacktoberfest challenge is run a workflow and fill out a brief survey about Relay.

In return we’ll send you a great Relay-themed t-shirt, inspired by the upcoming game, Cyberpunk 2077. The first 75 respondents will get a free t-shirt. We can confidently promise it won’t be as delayed as Cyberpunk. Sadly, we can only offer this shirt to those people living in the continental USA.

As you may already know, Relay (by Puppet) is an event-driven automation platform that pulls together all the tools and technologies DevOps engineers need to effectively manage their environment. It works by listening to signals from DevOps tools and apps people already use and then triggers workflows to orchestrate any required downstream service.

Here’s what you need to do to get your free t-shirt:

1. Sign up or login to Relay

If you don’t already have an account you can create one for free.

2. Run this workflow!

Install the workflow and then click the run button.

3. Click the “View Logs” button

In the logs you will find the URL for the survey. Head there to get started.

4. Fill out the survey

This should only take a few minutes.

5. We’ll send you a t-shirt!

Thanks for making our product really special and please see this as a thank you for your continued support.

How to Reduce MTTR with PagerDuty and Relay

Melissa Sussmann — Mon, 21 Sep 2020 00:00:00 +0000

DevOps and SRE teams are under intense pressure to reduce the Mean Time to Recovery (MTTR) in resolving incidents. With the proliferation of cloud services and the increasing complexity of DevOps toolchains, engineers today need to not only learn how to use these services but also troubleshoot them when an incident is raised at 2 AM. Incident response is still manual today – cobbling together runbooks and ad hoc scripts and orchestrating people to respond. This “digital duct tape” approach results in what we call the “DevOps Dumping Ground”, which ultimately extends MTTR.

How PagerDuty & Relay Work Together

PagerDuty is the industry-leading incident management platform that provides reliable notifications, automatic escalations, on-call scheduling, and other functionality to help teams detect and fix infrastructure problems quickly.

Relay by Puppet is an event-driven automation platform that pulls together all the tools and technologies DevOps engineers need to effectively manage a cloud environment. Unlike many existing workflow automation tools, Relay can intelligently respond to external signals by combining event-based triggers with a powerful workflow engine in a single platform.

The latest integration between Relay and PagerDuty eliminates the “digital duct tape” by creating reusable, event-driven workflows to close the loop on incidents faster through Relay’s event-based automation approach. PagerDuty users can now:

Enrich alert data : Using the new Change Events launched at PagerDuty Summit, Relay enhances alerts with diagnostic information to speed time-to-resolution by presenting more context around the alert.
Automate incident communication : Whether it’s creating a Slack room, updating a Jira ticket, or notifying team members, Relay ensures that communication is timely and updated.
Trigger Auto-Remediation Workflows : Raising PagerDuty incidents can initiate Relay workflow runs to fix troubleshoot & remediate common problems securely and quickly.

Example: How to Automate Incident Communication Plans

A key way to reduce MTTR is to formalize an incident communication plan. Making sure that teams have a robust plan for understanding roles and opening communication channels is key to reducing incident response time. Relay can automate this workflow for you by contacting the on-call engineer with a message detailing content from the incident.

Relay uses “triggers” and “steps” to automate a set of actions. Steps are reusable, modular, and composable – things like getting a user’s info, sending Slack and Twilio messages, and using the PagerDuty Event API to provide more information on an incident. “Triggers” are based on cloud events, git events, monitoring alerts, tickets, and incidents. In the example below, we see how a PagerDuty incident triggers the following incident response workflow utilizing the steps mentioned.

When a new PagerDuty incident is raised, Relay looks up the on-call person’s email address, identifies that user in Jira and Slack, and creates a Jira ticket for the production incident. Relay then creates a Slack room as a production incident command center, invites the on-call in, along with the pertinent engineering manager, and sets the topic of the room with a link to the Jira ticket that has been created. Finally, it sends a message to the Slack room and posts a note with the expectations of how a production incident policy should be followed.

Using PagerDuty’s exciting new Change Events, Relay elaborates on content from the incident with enriched alert data. This enables the individual on call to respond to the incident quickly, with less toil required for ticket creation and communication on what triggered the workflow.

Try out this workflow here.

Customize your Incident Response

There are several starter workflows available for PagerDuty users, which you can find on their integration page. You can use these workflows to create an issue in Jira, send a message to slack, and send a Twillo SMS automatically when a PagerDuty incident is triggered.

Everyone’s workflow is a little different, so Relay workflows are customizable for use cases. Relay provides contextual help within its sidebar. This feature lets you browse the library of integrations and steps to make it easy to customize your workflow.

Sign Up for Relay!

Use Relay with PagerDuty to reduce your incident response time and improve observability. Reducing your mean time to resolution (MTTR) is key to successful DevOps management and enabling event-driven automation will mean that your incident response time is much shorter. Relay makes this easier by using workflows that fix more common and well-understood problems that teams have already identified. To learn more about Relay, visit our site at relay.sh and sign up for our free beta!

The Event-Driven Web is Not the Future

Adam DuVander — Wed, 19 Aug 2020 00:00:00 +0000

When you see a notification on your smartphone, your brain processes the request quickly and determines how to react. It’s an efficient process and your nervous system is built for this use case. By contrast, most Internet-connected systems work in a less event-driven architecture. If there’s a change in one service, you won’t know about it until you check. It’s the equivalent of reloading an app to see if there’s something new—it works eventually, but it’s not efficient.

You might expect that the event-driven web should be the future. If systems knew about updates immediately, they could seamlessly make changes in reaction to the new information. New servers could be provisioned, unneeded resources could be turned off, and your microwave clock could always be accurate (ok, that might be asking too much).

The truth is: real-time patterns have been around for years. The evented web is not the future because the present is fully capable of what it offers. Most developers aren’t taking advantage of event-driven development. While the pieces are there, not every service supports events. Perhaps most importantly, there are few tools to easily consume events, because development is stuck in a client-server mentality.

As developers, it’s time to embrace this entirely un-new, but useful, approach to building Internet-connected systems.

What Real-time Patterns Can Accomplish

Whenever a change occurs in one system or new data is available in another, all of that context should be shared with systems that have declared an interest. In the same way that we expect smartphone notifications, developers can design for events. However, rather than causing more distractions for us to triage, they can save us time. Real-time patterns provide immediate updates without the manual button-pushing.

Consider the many tools required to build, deploy, monitor, and improve applications today. AWS itself has over 200 services and that’s just one cloud provider. Once you consider other cloud services and the ecosystem around all of it, you’ll be working with more tools than you’ll typically count, each with its own handful of API-driven knobs.

When those knobs are turned via thoughtful automation, you start to see what’s possible. You can streamline your deploy processes toward the promise of continuous delivery. You can trigger events based on pull request activity or system monitoring and scale up and down cloud needs in response to incidents.

Too often companies take event-based operations halfway. More instrumentation without automation is not the goal. Your team could very well spend all of its time dousing cloud-bourne fires. Each alert becomes a new task on a never-ending list. Even though we have the technology to reach the real-time opportunity, the momentum of how we’ve done things for decades holds us back.

We Are Stuck in a Client-Server Mentality

Since the early days of the Web, tools have operated on a simple model: a browser requests a resource and the server responds in kind. Front-end advances have given us interfaces that emulate real-time, but behind the scenes, these technologies often look a lot like the client-server model. It’s from that mindset that many of our tools and development processes are created.

If you’ve said “try reloading it” in recent memory, you recognize the issue. Servers respond to events, clients don’t. To move into the real-time present, servers must also send events, which means a client must be able to receive events.

There are current solutions to implement real-time patterns:

Polling , where you check for new data every minute or more
Webhooks , where you subscribe to receive updates as available
Websockets , a two-way protocol, and proposed standard

Each requires changes in how you architect your applications. You need to organize how you receive events, store the data, react to their contents, and chain the results to other services. Despite being an API-driven process, it’s unlikely to fit the model of your existing API integrations, which reside in the client-server mentality.

To break into the real-time paradigm requires tooling that supports the shift in thinking, without putting the additional architectural burden on your team.

The Best Teams Will Use Real-time Tools

The evented web, which allows for real-time patterns, is very much available now. You can bring its efficiency to your team if you organize the right tools. It is unlikely you’ll want to build the infrastructure yourself unless you have unique needs or a team of engineers waiting for their next project.

Some important features to look for when implementing real-time patterns:

Support for webhooks and polling
Integrations with the DevOps tools you already use
Audit trails of each run of the workflow
API secret management support

When we didn’t find anything to meet those needs, we built Relay. You can create automation with support for a growing number of DevOps and business tools. Write workflows in a familiar YAML syntax and run them in our secure environment. Try Relay for free now.

How to Provision Cloud Infrastructure

Adam DuVander — Wed, 12 Aug 2020 00:00:00 +0000

Provisioning Cloud Infrastructure

One of the best things about cloud computing is how it converts technical efficiencies into cost-savings. Some of those efficiencies are just part of the tool kit, like pay-per-use Lambda jobs. Good DevOps brings a lot of savings to the cloud, as well. It can smooth out high-friction state management challenges. Sprucing up how you provision cloud services, for example, speeds up deployments. That’s where treating infrastructure the same as workflows from the rest of your codebase comes in.

Treating infrastructure as code opens the doors to tons of optimization opportunities. One standout approach is standardization, which can simplify operational challenges. When you deploy from a configuration document, you decrease risk and speed up development. You also can employ those configuration files in automated DevOps workflows. In this post, we’ll give some examples of how you can leverage these benefits using Terraform for the deployment of cloud resources and Bolt for configuring them.

Deploy From Documentation

Terraform is great for building and destroying temporary resources. It can simplify an ad-hoc data processing workflow, for example. Let’s say you’re doing on-demand data processing in AWS. You need to spin up an EMR cluster, transform your data, and destroy the cluster immediately. This transient cluster workflow pattern saves you a ton. But manually deploying the cluster for each job slows down development time. With Terraform, you can write that cluster’s specifications once and check it into git to ensure you deploy the same version each time.

Terraform configurations are incredibly easy to write and read. They can also be easily modularized for reuse. Rather than plugging all of the configurations into one file, templatize the resource and the value for each argument from a tfvars file, which acts as a config.

Here is a truncated example of a templatized EMR resource that you might put in your main file.

resource "aws_emr_cluster" "cluster" {
  # required args:
  name = "${var.name}"
  release_label = "${var.release_label}"
  applications = "${var.applications}"
  service_role = "${var.service_role}"

  master_instance_group {
    instance_type = "${var.master_instance_type}"
  }

  core_instance_group {
    instance_type = "${var.core_instance_type}"
    instance_count = "${var.core_instance_count}"
  }
}

The vars are referenced from a terraform.tfvars file that inherits variable declarations from a variables.tf file.

terraform.tfvars:

name = "spark-app"
release_label = "emr-5.30.0"
applications = ["Hadoop", "Spark"]
master_instance_type = "m3.xlarge"
core_instance_type = "m3.xlarge"
core_instance_count = 1

variables.tf:

variable "name" {}
variable "release_label" {}
variable "applications" {
  type = "list"
}
variable "master_instance_type" {}
variable "core_instance_type" {}
variable "core_instance_count" {}

Notice how easy it is to modify an instance type. They’re all well-documented and centrally managed in the code. No one has to look up a Wiki or previous version of the application. Just check it out of git and refer to a single, deployable config. Note that this is an incomplete list of arguments. For a full list of optional and required arguments see Terraform’s aws_emr_cluster documentation.

Furthermore, by storing your Terraform repo in git, you can leverage event-driven automation workflows, such as redeploying the resource on merges into your master branch.

Automate Config Management

Now let’s look at how to conveniently update persistent infrastructure such as a fleet of always-on EC2 instances. Applying new provisioning actions to each one can be time-consuming. Bolt by Puppet helps you manage multiple remote resources at once. You can use it to perform scheduled uptime monitoring or you can run one-off patching tasks. In either case, Bolt tools can be captured within your projects and maintained in git. That allows you to apply the benefits of infrastructure as code to your configuration and maintenance programs.

Bolt actions are either tasks or plans. Tasks are on-demand actions. Plans are orchestration scripts. Let’s start with a simple task. Suppose your development team needs a Docker engine installed on a suite of EC2 instances. It would look like this:

bolt task run package action=install name=docker --targets my-ec2-fleet

The installation will be applied to all of the resources declared as targets in the projects inventory file.

Plans are declarative workflows written in YAML that run one or more tasks. That makes them easy to read and modify. A simple plan to provision newly deployed web servers with nginx would look like this:

parameters:
  targets:
    type: TargetSpec

steps:
  - resources:
    - package: nginx
      parameters:
        ensure: latest
    - type: service
      title: nginx
      parameters:
        ensure: running
    targets: $targets
    description: "Set up nginx on the web servers"

Notice that targets is parameterized. That allows you to dynamically apply a list of resources when the plan is executed. You can leverage that further by integrating Bolt with other DevOps workflows.

Consolidate Into a Workflow

Now we’ve covered provisioning with both Terraform and Bolt. Both are great tools that help you standardize infrastructure and configuration processes as code. You can even string them together in a modular event-driven workflow to reliably reuse and modify. Relay, a workflow automation tool from Puppet, provides integrations with Terraform, Bolt, and AWS. For example, declaratively map successful Terraform deployment as triggers that pass AWS resource IDs to Bolt for further configuration.

Check out other integrations and see how Relay can streamline your cloud provisioning workflow.

Take Control of your DevOps Dumping Ground with Relay!

Melissa Sussmann — Thu, 30 Jul 2020 00:00:00 +0000

click here for the webinar

Learn How to Use Relay to Clean up Your DevOps Dumping Ground

As the automation surface area grows to accommodate hundreds of interconnected APIs on the cloud, developers are using their own, home-grown “digital duct tape” to manage a growing “DevOps dumping ground”. For a lot of organizations, home-grown glue logic is inconsistent, not repeatable, and expensive to maintain hundreds of event-based workflows and thousands of combinations. We believe that the answer lies in automation workflows. In particular, workflows-as-code that can be triggered by events. We want to replace engineers’ home-grown digital duct tape with reusable, event-driven workflows.

In an effort to deal with ad-hoc deployments and devops infrastructure CI/CD management, many devs try to create their own one-off automation tools or integration hubs, usually per team or per project. Examples include:

Using Lambda and writing functions for EC2 management tasks
Running scheduled jobs for EBS cleanup
Repurposing a CI/CD tool like Jenkins for incident response workflows

But this “dumping ground” current approach is Inefficient, expensive and risky

Inefficient because work is done as one offs that last forever, no reusability or repeatability
Expensive because spending time building tools and integrations isn’t directly delivering customer value
Risky because sidestepping governance to get stuff done can lead to exposure and failures

This webinar presentation from Melissa Sussmann and Kenaz Kwa at Puppet gives viewers a peek into the beta version of Relay, a product for managing containerized apps. This presentation goes over what the team has learned in the process of working on Relay, the underpinnings of the product, and demonstrates a few example workflows to help you save time and money.

Why “No-Code” Tools are a Non-Starter for Developers

Adam DuVander — Thu, 16 Jul 2020 00:00:00 +0000

Will Developers Use “No Code” Tools?

Many attempts to simplify programming lead to visual interfaces that provide approachable settings for common tasks. These simplifications may appeal to non-developers, but they send experienced coders running for the command line. Yet, no code tools are rapidly expanding. Zapier, Integromat, and Workato are becoming more popular options and some believe developers won’t be needed for most integrations in the future. However, it seems unlikely that coders will adopt these tools in their current forms, as they are not looking to fully automate away their creative autonomy. This raises the question: which parts of currently available no-code tools are useful for the future developer? Furthermore, how can developers influence the production of these integrations so that they’re compatible with the true coder’s workflow?

Developers may be able to strike a balance between automating appropriate responses while still leaving space for them to impart creative nuance upon the end product. Certain patterns can be brought into a typical developer’s workflow which can streamline the way apps are built, deployed, and tuned.

Developers Don’t Want Drag-and-Drop Interfaces

Slick user interfaces with plug and play features do worlds to bring new layman users into the fold. However, these aren’t necessarily good for app developers. Modern coders want to be able to control their code down to the line, while still automating away repetitive maintenance processes. When drag-and-drop is a requirement, developers can’t have as much control or efficiency. As a result, the creative development process is interrupted and you’re unlikely to end up with the most innovative applications.

Tools like Scratch (and similar, more professionally-oriented, tools) are great for learning and simple projects. They are typically not granular enough in their controls to create meaningful business applications. And they usually slow down professional developers.

Developers thrive on the command line. They want to be efficient and automate away the stuff that keeps them from moving quickly. Part of what makes that possible is they can “see inside” and tweak things at a low level. Developers are likely to always want to design the components of their systems themselves, rather than dragging them out of a panel in a UI.

However, this is not to say there is nothing to learn from these no code tools. For example, repeatable workflows are useful if a dev can plug in their real code. This permits developers to write individual components of a stack, but with assistance from a system that automates the redundant parts.

Since much of the coding process is devoted to editing and retesting, there are plenty of pieces that can be streamlined. This becomes especially clear when you look at everything involved with backend maintenance.

Developers Don’t Want to Babysit Servers

No code integrations aren’t all bad. They can be extremely useful for developers when properly employed. They are a boon to company efficiency in two key cases: when they can save labor hours and when they can save server capacity. Developers of varying skill levels would surely benefit from automating much of their backend maintenance. Periodic functions could happen in the background without taking up valuable time coders could spend building.

Some unnecessary developer time stealers include:

Cost optimizing servers
Wiring up continuous integration
Connecting tools around incident response
Auditing cloud security permissions

Cost optimization is the elephant in the room for cloud computing. How companies acquire, build, utilize, and adapt their server space can hugely impact the efficiency of their spend. Currently, companies spend a boatload on labor hours for developers to monitor their systems.

It’s important to keep in mind that there are two kinds of jobs for developers, and indeed anyone selling a product: those that make your product more unique and routine functions which must be kept up in order to be a responsible administrator. Clearly, developers should focus as much of their energy and resources on advancing the company’s core value proposition. The less time developers spend getting distracted with mundane maintenance tasks, the better. This brings us to our next efficiency factor, minimizing interruptions.

Developers Do Want to Automate Their Interruptions

A tap on the shoulder. Expanding Slack notifications. Unnecessary pages for non-incidents. These are some of the things that keep developers from doing their best work. These can’t all be avoided, but automation can help limit them.

For example, a developer might get a notification when their cloud development servers have been running idle for too long. Those messages are well-meaning, but they are a distraction from focused work. The actions taken during these times may only take minutes, but then it takes time to get back into a productive flow. Perhaps most maddeningly, the actions needed here are always likely to follow the same considerations. It’s a perfect opportunity for automation.

Or consider code review, an important part of working on a dev team. The reviewer should not need to manually create a staging server with running code. Nor should they need to worry about shutting it down when they’re finished. Between continuous integration tools, code repositories, and a way to describe the ideal flow, you can limit the time a developer is interrupted.

The rise of “no code” gives development teams an opportunity to look where they’re wasting developer time. The answer is not to give drag-and-drop interfaces to developers. Let them use the tools they know best and connect those pieces with real code.

Using a combination of event-based triggers and automated protocols, Relay listens to signals from your existing DevOps tools and then triggers workflows to orchestrate actions on downstream services. Developers can get a taste of no code without giving up their code. Be efficient with the things that can be automated and get your team to get back to building out your organization’s core value.

Get started today with our single platform for all your cloud automation use cases.

Relay Public Beta

Deepak Giridharagopal — Wed, 24 Jun 2020 00:00:00 +0000

Today we announce Relay, an event-driven automation platform. Sign up now and try it out! Relay connects infrastructure and operations platforms, APIs, and tools together into a cohesive, easy-to-automate whole. Relay is simple enough for you to start automating common, if-this-then-that (IFTTT) style DevOps tasks in minutes and powerful enough to model multi-step, branching, parallelized DevOps processes when the need arises.

Why bother? Because for all the progress we’ve made as builders and operators, things are more complicated than ever. Modern applications comprise a growing variety of runtimes, clouds, infrastructure platforms, 3rd party services, and APIs. Mounting sophistication (and complexity) of how applications are constructed complicates how we operate and manage them. As a result, accomplishing many basic operational tasks can involve touching many different components, with different APIs, different semantics, from different upstreams, vendors and dev teams. Connecting all of these components together is tough, and automating anything across them all can range from tedious to nightmare fuel.

At layers above the plumbing, managing infrastructure stops looking like classic configuration management and starts looking like orchestrating workflows. However, workflows can be tricky. Connectivity, secrets handling, event listening, ordering, parallelism, error handling, and control flow all conspire to make writing workflows from scratch pretty gnarly. The complexity adds up fast. We can do better!

“Automated workflows are the bedrock of all software organizations.”

— Jason Warner, CTO @ GitHub

Relay lets you represent any DevOps workflow as code, composed of triggers that listen for incoming events, and steps that define the task you’re automating. Relay does not limit what you can talk to. A single workflow could listen for alerts from PagerDuty, query metrics from DataDog, reconfigure infrastructure with Terraform, and send a notification via Slack. It’s easy to leverage pre-existing triggers, steps, and workflows, and it’s simple to make your own if the need arises.

As a hosted service, Relay supervises things on your behalf. It will automatically trigger your workflow based on incoming events, execute your workflow’s steps in parallel, notify you if you need to intervene, and keep meticulous records of everything done. Relay does this all automatically, so you don’t have to.

Today, we’re proud to announce beta availability for Relay. Read on to see how it works!

Workflows

Relay’s core method of automation is the workflow. Workflows combine useful activities together to accomplish a particular task:

When we detect an unused Azure Disk, delete it (so we can save money)
When they go unused, nuke any AWS authentication keypairs (so we can reduce our attack surface)
When a PagerDuty alert fires with a certain severity, create tickets in Jira and a room in Slack (so we can more quickly troubleshoot issues)

Relay lets you succinctly express these types of workflows, and beyond, as code. And like code, workflows can be versioned, reviewed, refactored, and reused. We’re Puppet; we wouldn’t have it any other way.

Running your first workflow is easy, and should only take you about a minute. As tradition demands, here’s “Hello, world” (log in and follow along!):

Because workflows are code, you can treat them like code. Modifying a workflow is straightforward. Let’s change the workflow, adding a step to emit the current date:

That covered getting the CLI installed, authenticating against the service, downloading your workflow, modifying the logic, and then letting Relay know the code is updated.

Triggers and steps

Workflows contain triggers and steps: Triggers determine when Relay should execute your workflow: manually, on a schedule, or when pinged by an external source. Steps represent the set of actions and activities necessary to make your workflow accomplish its goals. Steps are just containers, so you’re pretty unconstrained when it comes to what a step can do. Both triggers and steps are easy to create, remix, and share. With these building blocks, Relay is capable of modeling a huge variety of workflows, and executing them on your behalf. There are a bunch already written, and it’s straightforward to make your own.

Here’s a more interesting example workflow that cleans up some unneeded EC2 instances. It has more steps, including some that consume AWS credentials, and one which represents a manual approval gate:

It’s easy to add, remove, or replace triggers and steps to suit your liking. Some modifications for the preceding example could include adding a webhook-based trigger for the workflow, adding a notification step at the end, or better integrating it into your GitOps setup. Perhaps a step that takes the list of terminated instances, computes their money you just saved, then buys an equivalent amount of stuff from your Amazon wish list? Ops is hard work - treat yourself!

Our examples thus far have shown short, linear sequences of steps but you can also express some pretty elaborate processes just as easily. Here’s a picture of the execution graph of one of workflows we use to manage Relay itself:

The Relay service

Listening for events and running workflows might appear conceptually simple, but there are a lot of practical details that need to be worked out. Relay’s execution environment (and underlying engine):

Manages connections to upstream/downstream APIs and services, making them securely available to the workflows that need them
Automatically creates push triggers for your workflows, complete with workflow-specific security tokens, so you can easily kick it off from all kinds of other tools
Automatically constructs an environment for running webhooks, so your workflows can respond to events from webhook-only services
Sandboxes workflow and step execution, for fault isolation
Manages your workflows with all the necessary ops accoutrements (e.g. monitoring, logging, error handling)
Supervises the execution of your workflows, invoking steps in the right order (with automatic parallelization)
Standardizes the interfaces between all these pieces so steps, triggers, and connections are reusable and remixable across workflows

Relay takes care of this stuff so you don’t have to. Instead, you can focus on the logic of your workflow, the core of what you’re trying to automate.

After all, isn’t that the point?

Automation for everyone

How many unique applications are running out there across the planet (or above it)? Thousands? Millions? Bajillions? How many of them are running on identical infrastructure stacks, built with identical technology stacks, managed in identical ways at an identical scale? There’s a truly staggering variety of approaches and constraints.

If there’s no One True Stack, then there’s no One True Way To Manage It. The tools you employ should thrive in this sort of world because that’s the world we’ve got.

Relay’s core value lies in letting you tie a wide variety of services, APIs, and platforms together. It’s constructed in a deliberately pluggable way. Users can readily extend the system to talk to new technologies, respond to new kinds of events, and take action in new ways…no CS degree required. Those extensions should be easy to share, so the entire user community can benefit. The ecosystems around the tools we use are every bit as important as the tools themselves.

Even though it’s early days, Relay can already do quite a lot. The future holds many possibilities: new workflows, more integrations with more tools and platforms, higher-level workflow syntax, a more streamlined authoring experience, simplified input/output from steps, and more. Early users have already given us a ton of great suggestions, and we’d love to hear yours!

Next steps

The next step (and best step) is to try it out! And if you’d like to learn more about Relay, you can check out:

How to get involved, extend Relay to better meet your needs, and become part of Relay community
The documentation does a great job of introducing Relay, its usage, core concepts, and extension points
Peruse some workflows to see what they can do. The code and its graphical execution plan are available on every workflow’s page.
Slack - come linger in the #relay channel! The more the merrier

Thanks, and let a thousand workflows bloom!

User-defined Webhooks in Puppet Relay with Knative and Ambassador API Gateway

Noah Fontes — Tue, 23 Jun 2020 00:00:00 +0000

User-defined Webhooks in Puppet Relay with Knative and Ambassador API Gateway

This article is a technical deep dive originally published on the Ambassador blog.

What is Puppet Relay?

Relay is an event-driven automation platform designed to make wrangling diverse operational environments easy. Relay executes workflows, which consist of multiple related steps, to perform actions like opening Jira tickets, merging pull requests, or even deploying an application to a Kubernetes cluster.

Relay is built on containerization and leverages Tekton to execute workflows. Each step in a Relay workflow runs an OCI-compatible container image. Unlike conventional workflow automation tools, this gives you the ability to make your own completely custom steps; you’re not restricted to our curated steps or even to a particular programming language.

When we set out to implement triggers to automatically run workflows from event data, we wanted to make sure you’d have similar flexibility as within a workflow to receive external data. We decided to provide three initial options: schedule triggers, which run your workflows on a specified interval; push triggers, which allow your services to directly send data to Relay; and webhook triggers, which integrate with lots of external services that can push event data to arbitrary HTTP endpoints.

Webhook triggers presented the biggest technical challenge to implement, as every service provides slightly different payloads representing their events. Keeping with our container-based approach, we let you define webhook triggers by running your own web server in a container you provide to us. In this post, we’ll walk through how we implemented our webhook trigger handling using Knative Serving and the Ambassador API Gateway.

You can see an example of a complete webhook trigger implementation in our PagerDuty integration.

Installing Knative Serving

We chose Knative Serving as a reverse proxy for webhook triggers. We could have configured a Kubernetes deployment for each webhook trigger, but we felt the resource burden on our cluster would be too high to have every customer pod running all the time.

Knative Serving’s unique model uses an activator and autoscaler to dynamically provision pods when it receives requests. A rough state machine for a Knative service looks like:

When a service is created or updated, a revision is created initially in an inactive state. This corresponds to a Kubernetes deployment scaled to zero pods. In this inactive state, HTTP requests are routed to the Knative activator.
When an HTTP request is received for the service, the Knative activator queues the request and switches the current revision to an active state. Knative then scales the deployment up and switches the service to route to the deployment’s pods. Once the service is pointing at the deployment, the queued HTTP request is dispatched.
The Knative autoscaler monitors inbound request rate and scales the deployment as needed.
If the service doesn’t receive any HTTP requests after a configured timeout, the deployment is scaled back to zero, the revision is marked as inactive, and the service is switched back to the activator.
Repeat from the beginning!

For a cached container image, the whole activation process generally takes less than 2 seconds, quickly enough for our webhook handling use case. And for webhook triggers that only receive events every few minutes or less frequently, it saves us considerable cluster resources.

Installing Knative Serving is straightforward. You need their CRDs and core components like the activator. Here we’ll use version 0.13.0, but check their installation instructions for the latest version.

$ kubectl apply -f https://github.com/knative/serving/releases/download/v0.13.0/serving-crds.yaml
$ kubectl apply -f https://github.com/knative/serving/releases/download/v0.13.0/serving-core.yaml

Then you need to pick a networking layer, or gateway, to route requests.

Choosing a Knative Serving Gateway

Like most Knative Serving users, we started by evaluating Istio, a popular service mesh and ingress gateway offering for Kubernetes. However, Istio’s focus on connecting microservices didn’t really support our use case:

We didn’t need the service mesh component of Istio at all, only its Envoy gateway. A complete Istio installation on our cluster consumed a lot of resources, most of which were ultimately going to waste.
Because we’re configuring webhook triggers dynamically from our database, we put our own reverse proxy in front of Knative Serving. It is difficult (although not impossible) to change the behavior of Istio to run as an internal-facing service instead of a public gateway.

Most other networking layer options for Knative Serving were positioned more strongly as ingress gateways, focusing mainly on exposing services directly to the internet.

Ultimately we settled on Ambassador because its lightweight single-container model made deploying it for our internal use case easy.

Installing Ambassador and Configuring Knative to Use it

For Relay, we use a custom Helm chart to set up the Ambassador API Gateway. We have a single deployment, an optional horizontal pod autoscaler, a service account with corresponding role bindings, and finally a ClusterIP service to use as the target networking layer for Knative.

Our deployment YAML is largely the same as the one from the Ambassador installation instructions in ambassador-rbac.yaml. However, we must explicitly enable Knative support:

# ...
template:
  spec:
    containers:
    - image: datawire/ambassador:1.5.2
      env:
      - name: AMBASSADOR_KNATIVE_SUPPORT
        value: 'true'
      # ...

Likewise, our service account and role bindings are similar to those in ambassador-rbac.yaml, but use {{ .Release.Namespace }} instead of default.

For the gateway service, note especially:

The app.kubernetes.io/component label must be present and set exactly to the value ambassador-service or Ambassador won’t pick it up to set the Knative service target correctly.
We use type: ClusterIP to make the service cluster-local. This instance of Ambassador won’t be reachable from the internet.

{{- $name := include "ambassador.name" . -}}
{{- $fullname := include "ambassador.fullname" . -}}
{{- $namespace := .Release.Namespace -}}
apiVersion: v1
kind: Service
metadata:
  name: {{ $fullname }}
  namespace: {{ $namespace }}
  labels:
    app.kubernetes.io/name: {{ $name }}
    # Per the Ambassador source code, this must be specified explicitly exactly
    # like this.
    app.kubernetes.io/component: ambassador-service
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/version: "{{ .Values.image.tag }}"
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    helm.sh/chart: {{ include "ambassador.chart" . }}
spec:
  type: ClusterIP
  ports:
   - port: 80
     targetPort: 8080
  selector:
    app.kubernetes.io/name: {{ $name }}
    app.kubernetes.io/instance: {{ .Release.Name }}

As a reference, you can find our entire Ambassador Helm chart on GitHub.

Finally, we need to configure Knative to use cluster-local routing and Ambassador as its default gateway. Simply apply this manifest to your cluster using kubectl apply:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-domain
  namespace: knative-serving
data:
  svc.cluster.local: ''
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-network
  namespace: knative-serving
data:
  ingress.class: ambassador.ingress.networking.knative.dev

Deploying, Testing, and Managing Internal-Only Knative Services

Now you can create a cluster-local Knative service. Use kubectl apply on a manifest like this:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-test-service
  namespace: default
  labels:
    serving.knative.dev/visibility: cluster-local
spec:
  template:
    spec:
      containers:
      - image: gcr.io/knative-samples/helloworld-go
        name: helloworld-go

Within a few seconds, Ambassador will process the service and set up a mapping for it. Assuming you installed Ambassador to the ambassador namespace, you’ll see this service when you run kubectl get svc:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-test-service ExternalName <none> ambassador.ambassador.svc.cluster.local <none> 112s

If you don’t see the service pointed at your Ambassador deployment, inspect the Knative service and ingress using kubectl describe ksvc my-test-service and kubectl describe king my-test-service. The status conditions and events should provide useful hints to remediate any problems.

Now we can try sending a request to the service by running a one-off pod:

$ kubectl run \
    --generator=run-pod/v1 internal-requester \
    --image=alpine --rm -ti --attach --restart=Never \
    -- wget -q -O - http://my-test-service.default.svc.cluster.local
Hello World!
pod "internal-requester" deleted

Yay! Your service works. You’ll also see a pod running to handle the request you just made. If you don’t make any more requests, within a few minutes, that pod will be automatically terminated.

NAME READY STATUS RESTARTS AGE
my-test-service-cw9v2-deployment-59c889f74b-74rwk 2/2 Running 0 8s

You can view all the mappings Ambassador has configured for your Knative services by forwarding the admin endpoint of your deployment:

$ kubectl port-forward -n ambassador deployment/ambassador 8877
Forwarding from 127.0.0.1:8877 -> 8877
Forwarding from [::1]:8877 -> 8877

Then navigate to http://localhost:8877/ambassador/v0/diag/.

For Relay, we manage our Knative services by writing out a higher-level CRD that our operator processes. This lets us perform lifecycle management operations more efficiently. For example, we create and clean up webhook triggers and workflow runs in batches we call tenants. We get a ton of value from the combination of Tekton, Knative, Ambassador, and our own operator, with relatively little cluster resource overhead.

Deployment Scenarios for Ambassador and Knative

The developer use cases for Knative fall into three categories:

Replace glue/aggregation functions with Knative

Function as a Service (FaaS) offerings have become popular as a way to deploy and run services that “glue” functionality together. The main challenge for development teams is that the workflow for deploying cloud-based FaaS is different than that for Kubenetes. If you’ve already invested in training engineers to work with Kubernetes, then the added time and money to train them is extraneous.

Build smaller microservices as functions

Some simple functions are event-driven and provisioning (or running) an entire microservice/app framework seems unnecessary. Knative provides “just enough” framework to deploy and manage the lifecycle of a very simple microservice or “nanoservice” using the primitives provided within modern language stacks.

Deploy high-volume functions, cost-effectively

Pay-as-you go serverless offerings can be very cost-effective for certain use cases. For longer-running or high-volume functions, pay-as-you-go isn’t as practical. Running Knative on your own hardware, or even running this via Kubernetes deployed on cloud VMs, can enable the easier costing of execution when you know that you will be running a service that has high-volume traffic.

Summary

Running on-demand microservices in your infrastructure is easier now than ever before. With the help of Knative and Ambassador, you can drastically reduce costs and resource utilization while maintaining clean separations of APIs across your environment.

At the frontier of cloud-native experience, Knative also unlocks some very exciting opportunities that haven’t been practical in conventional deployment environments. In this post, we explored low-trust user-defined workloads using custom containers as one example, but there are many more!

Next Steps

Install the Ambassador Edge Stack.
Install Knative with Ambassador.
Read the documentation for using Knative and Ambassador.
Contact the Ambassador team to learn more about using Ambassador with Knative.
Sign up for Relay to try webhook triggers yourself.