DEV Community: Marko Anastasov

An “n8n for DevOps” Control Plane - Superplane

Marko Anastasov — Tue, 27 Jan 2026 07:23:17 +0000

We’re launching Superplane, an open-source control plane similar to n8n that orchestrates your entire DevOps workflow.

If this sounds interesting, please give us a star on GitHub ⭐ it helps us grow the project.

The Problem Superplane is Tackling

Deploying to production is rarely a straightforward process.

You’ve got CI/CD pipelines in to manage, release management in Jira, incident response in PagerDuty, monitoring in SigNoz, notifications in Slack. And somewhere in between, you’re manually coordinating all of them.

You can’t simply write a script that will handle all the cases of a complex workflow, where one thing has to happen after this other thing is done running. Creating a mental map of these workflows is also no easy task.

DevOps workflows are fragmented across a dozen tools, and we built the first control plane to orchestrate them all. Yes, we were inspired by n8n a lot.

The “n8n for DevOps” Vision

If you’ve used n8n, you know the magic: visual workflows, event-driven triggers, integrations with everything.

But here’s the thing, n8n is built for general automation. It’s amazing for connecting your CRM to your email, or syncing data between APIs. But when it comes to DevOps processes like deployments and progressive rollouts, you still have to manage custom nodes, states, and hope your webhooks don’t timeout.

What if there was an n8n specifically for DevOps?

A platform where you define your operational workflows once, and they just… work.

With Superplane, you define your services, environments, releases, incidents, and signals in one place. This way it’s easy to understand how your system behaves and what the exact workflow is. The tribal knowledge of your DevOps team gets captured.

We’d love it if you could please give us a star on GitHub ⭐ as it helps us sustain the project.

What Makes Superplane Different

Superplane isn’t trying to replace your CI/CD, your monitoring, or your incident management tools. Instead, it’s the control plane that orchestrates them.

Event-driven Workflows

Superplane listens to events from your existing tools:

CI pipeline completes → trigger workflow
Schedule hits → run maintenance task
Webhook arrives → process and route
Alert fires in PagerDuty → start incident response

You define workflows visually (like n8n), but with DevOps-specific primitives built in.

Operational Context

Unlike generic automation tools, Superplane understands DevOps concepts:

Approval gates: Require human approval before proceeding
Policy checks: Enforce business hours, deployment windows, team policies
Progressive rollouts: Deploy in stages with verification between each
Multi-service coordination: Wait for multiple repos/services, then fan-in

Self-Hosted, Open Source

Superplane runs on your infrastructure. You own your workflows and data, everything is tailored to self-hosting needs.

After months of assembling the team and building this from ground up, we’re finally ready to share Superplane with the world. We’re launching early, we’re moving fast, and there are rough edges. But we believe it’s ready for teams to start using it across different organizations.

Reliability and State Management

Every DevOps engineer reading this must be thinking: "This sounds nice, but what happens when things go wrong?"

These are the hard problems in orchestration, and we've built Superplane to handle them.

Durable State Persistence

Every workflow execution is stored in PostgreSQL. Events flow through the system with tracked states (pending → routed → finished), and each node execution records its input, output, and status. If Superplane restarts, nothing is lost, it picks up where it left off.

Configurable Retries for HTTP Requests

The HTTP component supports:

Configurable retry count: Set how many times to retry failed requests
Two timeout strategies: fixed (same timeout each attempt) or exponential (timeout doubles each attempt, capped at 120 seconds)
Retry metadata tracking: Each attempt records its status, so you can see exactly what happened

End-to-End Event History

Every event and execution is logged with full context like inputs, outputs, timestamps, and state transitions. When something goes wrong, you can trace exactly what happened, when, and why.

Inspectable Workflow Runs

The control plane UI lets you inspect runs, see status at each step, and understand where things are stuck or failed. No more digging through logs across 5 different systems.

Real Workflows for Running Superplane in Production

Here's what teams are actually building with Superplane:

Policy-Gated Production Deploy

CI finishes green
  → Check: Is it business hours? (No → hold)
  → Check: Is on-call engineer available? (No → notify and wait)
  → Require: Product manager approval
  → Trigger: Production deployment
  → Verify: Health checks pass
  → Notify: Team in Slack

All in one workflow. No manual coordination needed.

Progressive Delivery

Deploy to 10% of traffic
  → Wait 5 minutes
  → Check: Error rate < 0.1%? (Yes → continue, No → rollback)
  → Deploy to 30% of traffic
  → Wait 5 minutes
  → Check: Error rate < 0.1%? (Yes → continue, No → rollback)
  → Deploy to 60% of traffic
  → Wait 5 minutes
  → Check: Error rate < 0.1%? (Yes → continue, No → rollback)
  → Deploy to 100% of traffic

Automated progressive rollout with automatic rollback on failure.

"First 5 Minutes" Incident Triage

Incident created in PagerDuty
  → Parallel: Fetch recent deploys (last 2 hours)
  → Parallel: Fetch health metrics (error rate, latency)
  → Parallel: Check recent code changes
  → Combine: Generate evidence pack
  → Create: GitHub issue with context
  → Notify: On-call engineer with summary

Automated incident response that gives you context before you start debugging.

How to Run Superplane

Superplane is self-hosted. You can run it on a single host or on Kubernetes.

Quick start:

docker pull ghcr.io/superplanehq/superplane-demo:stable
docker run --rm -p 3000:3000 -v spdata:/app/data -ti ghcr.io/superplanehq/superplane-demo:stable

Then open http://localhost:3000 and start building workflows.

Production installation:

Single Host Installation - Deploy on AWS EC2, GCP Compute Engine, or other cloud providers
Kubernetes Installation - Deploy on GKE, EKS, or any Kubernetes cluster

🚀 Try Superplane Now

Superplane is open source (Apache 2.0), built by developers, for developers. No vendor lock-in. No per-seat pricing. Our workflows simply work.

👉 Star us on GitHub to support the launch and stay updated!

Join the community: Discord | X | Website

You Need To Change The Whole Assembly Line

Marko Anastasov — Wed, 21 Jan 2026 12:59:52 +0000

In the DevOps Paradox episode 2026 — The Year of Discovery, Viktor and Darin talk about how software work is changing, but the bottlenecks aren’t where people think.

A few quotes are basically a version of why we started SuperPlane. My comments below.

You cannot just give developers, here’s AI. Everything else stays the same… you need to change the whole assembly line.

Most organizations are currently applying AI to optimize the coding process.

The constraint will simply move downstream: releases, verification, coordination, incident handling.

If you don’t address the system, you get faster queues, not faster outcomes.

You have a platform that you can say, hey, give me a database, and you get that database. And that database is done based on your company's best practices, rules, policies [...] ask me questions that I need to answer and don't ask anything more.

This is what engineers are actually asking for: outcomes, and systems that hide incidental complexity without hiding state.

People simply don’t want to… relinquish part of their power… [and] don’t want to introduce randomness in production.

This is the hard part. It’s not so much a capability problem as much it's a trust problem. People will only use automation in production when it’s controllable, auditable, and fails in predictable ways.

To get there, I think we need a control plane that makes cross-tool operational workflows explicit and inspectable: what triggered, what ran, what it touched, and what evidence it produced. If you can’t explain a change, you can’t safely automate it.

Cloudflare's November 18 Outage – A Continuous Delivery Perspective

Marko Anastasov — Thu, 27 Nov 2025 16:19:03 +0000

On November 18th, 2025, Cloudflare had what they describe as their worst outage since 2019.

It didn't start with a cyber attack or massive hardware failure. It started with a database permissions change.

This is exactly the sort of thing Continuous Delivery is supposed to make safe. So since the post-mortem came out I wanted to analyze it through a CI/CD lens.

Summary of what happened

Around 11:20 UTC, Cloudflare’s network started returning a huge spike of HTTP 5xx errors for traffic flowing through their core network. Users all over the Internet started seeing Cloudflare error pages instead of the sites they were trying to visit.

The root cause was a change to how one of their ClickHouse database clusters handled permissions and metadata.

Here’s the rough chain of events:

Cloudflare deployed a change to ClickHouse permissions and metadata visibility. That change meant queries against system metadata suddenly started returning more rows than before — including underlying tables in another schema (r0) where previously they only saw the default database.
One of the systems that depended on that metadata was Cloudflare’s Bot Management feature file generator. It queried system.columns to assemble the list of features used by a machine learning model to score bots. That query didn’t filter by database name, so after the change it suddenly saw roughly double the number of columns, and produced a much larger feature file.
That feature file is shipped out to Cloudflare’s edge network every few minutes. The bots module in their core proxy expects the number of features to be below a hard limit (200) and preallocates memory accordingly. The new, larger file exceeded that limit. The Rust code hit the limit and panicked, leading to 5xx responses from the core proxy.
Because this file is generated every five minutes from a distributed ClickHouse cluster that was being gradually updated, the system actually went up and down for a bit: sometimes a good config, sometimes a bad one, depending on which node produced it. Eventually all nodes produced the bad file, and the failure became stable.
Downstream systems that depend on the core proxy – things like Workers KV and Access – were also impacted until they were temporarily rerouted or bypassed.

Main impact started around 11:28, substantial mitigation began around 14:30, and everything was fully resolved by 17:06 UTC.

So: a change to database permissions → bigger config file → hard limit exceeded → proxy panics → global outage.

That’s a change management story.

This is why Continuous Delivery exists

The primary cause of production failures is change. Continuous delivery is about making change boring.

If your system is updated frequently then you must assume that any change is guilty until proven innocent.

From a CD perspective, a few big themes jump out:

1. Config is code

The feature file that broke things isn’t some harmless data; it is executable behaviour. It changes what the bot-scoring model does and it clearly has operational consequences. That should be treated with the same discipline as application code:

Stored and versioned in source control
Validated in pipelines
Subject to contract tests with the consumers (the proxy module)

2. Contracts between systems

There is an implicit contract here:

Producer: “I will generate a feature file.”
Consumer: “I can safely handle up to 200 features.”

That contract wasn’t explicitly enforced at integration boundaries. The producer didn’t know that exceeding 200 features would crash the consumer, and the consumer effectively said: “If this assumption is broken, I will simply panic.”

In a good CD setup, we encode these contracts as tests:

“If the feature file has > 200 features, the pipeline fails.
Or, at runtime: “If it’s too big, log, drop extra features, and degrade gracefully instead of crashing.”

3. Fast feedback tied to change events

When production starts returning loads of 5xxs right after a change, your first hypothesis should almost always be: “We broke it.”

Cloudflare initially suspected a large DDoS attack, partly because their off-platform status page also went down by coincidence, which complicated the diagnosis.

A strong CD/operational posture ties monitoring directly to deployments:

“We deployed change X to ClickHouse at 11:05.”
“At 11:20, 5xxs spike globally.”
The system should scream: “Roll back that change first, then keep investigating.”

To their credit, they did eventually correct course, stop propagation of the bad file, push out a good one, and restart the proxies.

Where CI/CD could have helped more

Let’s talk about specific points where mature CI/CD practices can help avoid or limit this sort of outage.

1. Test the config generator like production code

The change that triggered this was essentially a behaviour change in the query used to build the feature file.

In a CD world, we’d treat that as an ordinary code change and ask:

Do we have automated tests for the query behaviour?
Do we have tests that assert the range of output sizes we consider safe?
Do we have a test environment where a realistic ClickHouse cluster – including these permission changes – is exercised before we touch production?

A property test like:

Given realistic metadata, the resulting feature file must have ≤ 200 features, or the build fails

...would have caught this before it ever reached live traffic.

2. Practice progressive delivery for everything

Cloudflare already had a gradual rollout of permissions across the ClickHouse cluster, but the effect of that was system going up and down: sometimes a good config, sometimes a bad one.

From a CD perspective, you want controlled blast radius:

Ship new behaviour to a tiny slice of traffic or a subset of regions first.
Observe: “Did the new feature file cause any spike in 5xxs, latency, or resource usage?”
Only then ramp up.

Instead of having the feature file immediately pushed to the entire network every five minutes, imagine:

A canary group of edge nodes is updated first.
If they see the bot module panic or error rates spike, an automated system:
- Rolls back to the last known good config file.
- Blocks further rollout.
- Raises an incident with a clear “config rollout blocked” signal.

That’s progressive delivery applied not just to code, but to ML feature sets and configuration.

3. Design for graceful degradation, not panic

The Rust code in the affected FL2 proxy is essentially designed to "panic" (as per post-mortem) if it gets more than 200 features.

From a resilience standpoint, that’s exactly what we don’t want. In a world of continuous delivery and constant change, you assume your assumptions will be broken.

Better options might include:

Drop any features over the limit and log loudly.
Disable the Bot Management module temporarily and continue forwarding traffic, maybe treating everything as “unknown bot score” rather than bringing down the proxy.
Trip a feature kill-switch that turns off Bot Management globally while keeping the core CDN and proxy path alive.

Cloudflare’s own remediation list mentions:

Hardening ingestion of internal configuration files like user input.
More global kill switches for features.
Reviewing error handling across core modules.

That’s exactly the direction you’d expect from a team embracing CD and resilience engineering.

4. CI as safety net for "platform-ish" changes

One subtle thing here: this wasn’t a direct change to Bot Management. It was a change to how the database platform handled permissions and metadata.

In many organisations, platform changes are treated as “infra stuff”, not subject to the same product-level tests. But in a CD culture:

Platform changes go through pipelines that also run representative workloads and integration tests for the services that depend on them.
That ClickHouse permission change should have run:
- A compatibility test suite for all consumers that query system.columns.
- Specific tests for the Bot Management feature generator pipeline.

If you can’t do that comprehensively, you at least start with the critical systems: anything that can bring down your main proxy path should have extremely strong automated protection.

Good engineering is about turning failure into fast, safe learning

Cloudflare did the right thing by publishing a detailed post-mortem, while expressing engineering humility and laying out concrete follow-up steps.

Everyone running large-scale systems has failures. What distinguishes good engineering organizations is:

They treat incidents as opportunities to improve the system, not to blame individuals.
They share what they learned.
They make structural changes – to code, to pipelines, to architecture, to daily practices.

From a continuous delivery perspective, the lessons I’d highlight are:

Every change is potentially dangerous. CD is about making lots of small changes safe, not moving fast and ignoring risk.
Config, queries, and ML features are code. They need the same CI/CD discipline: tests, contracts, and progressive rollout.
Design for graceful failure. When – not if! – your assumptions are broken, the system should bend, not snap.
Tie observability tightly to deployments. If you’ve just changed something and the world is on fire, suspect your change first.

Cloudflare's outage is painful for them and for a lot of the internet. But it’s also a rich example of why we do continuous delivery in the first place.

Stop calling bugs 'major' and 'minor'

Marko Anastasov — Mon, 15 Jul 2024 15:22:11 +0000

I recently cleaned up Operately's issues (now all public). Split them into bugs and papercuts.

I've seen many teams use "major", "minor", etc for bugs. But what does that really mean? We prefer more specific terms.

There's a big difference between:

❌ Something that should work but doesn't
🚧 Something that's just unfinished or crude

Calling every little issue a "bug" in a new product is demotivating.

Hence, papercuts.

Papercuts are small annoyances. Things like:

Usability quirks
UI inconsistencies
Any little thing that frustrates users

Like, "posting a comment on a goal update should generate a news feed item". Sure, it should. But we never implemented it. We will get to it, though.

On their own, they seem trivial. But they add up fast.

So of course, in theory you want to fix all your papercuts. Your product will feel smoother. Users will be happier.

In reality, Operately is still in alpha stage and we have to balance feature completeness with a high enough quality to meet initial user expectations and validate it in the market.

So for now the strategy is, we'll trim them to the level that we estimate is acceptable.

If you read this this far 🫶 -- how do you handle bugs and papercuts in your projects?

My Aha! Moment with Test-Driven Development

Marko Anastasov — Thu, 25 Nov 2021 13:19:27 +0000

My aha! moment with test-driven development were acceptance tests and I recommend everyone who is still a skeptic to start there.

The Benefits of Acceptance Testing

Tomas Fernandez for Semaphore ・ Nov 18 '21

#testing #beginners #cicd #programming

Many years ago I was working on a Ruby on Rails app with a multi-step booking process. Each step had many possible states and outcomes.

Testing that manually after every change in code was painfully repetitive, slow, and embarrassingly unreliable. 😓

Getting anything done was hard. We’d change one thing, then spend about an hour on manual testing in the web browser to check if everything is still working.

We’d often be surprised by the fact that we broke a seemingly unrelated use case. 🤯

Sometimes we’d figure that out on our own, and sometimes we’d ship a new version to production and the client would email us about what had just stopped working.

Getting that kind of news felt the worst. It felt like we were wasting a lot of our client’s and our own time. 👎

Then, we discovered Cucumber.

Very easy to get started. We could write a test once, and it would automatically launch a web browser, run the application, and do the work of feature verification instead of us.

That felt like magic! 🪄 ✨

As soon as we started writing acceptance tests for a feature we had been working on, our development accelerated by an order of magnitude. 🚀

We could add and change code without worrying that we would break something. 😌

When something did break, a failing scenario would let us know, and we’d quickly fix it before deploying a new version to users. 🦄

So to all teams out there who don't write (much) tests, I Hundred points symbol recommend starting with acceptance tests.

The ROI — in peace of mind, reliability, velocity, and good feeling — of writing just a few is massive.

From that point on, it's really a downhill ride to adopting a test-driven mindset for all layers of code. 💪

Originally published as a Twitter thread.

How to Test Jupyter Notebooks with Pytest and Nbmake

Marko Anastasov — Thu, 08 Jul 2021 16:03:06 +0000

Jupyter notebook files have been one of the fastest-growing content types on GitHub in recent years. They provide a simple interface for iterating on visual tasks, whether you are analyzing datasets or writing code-heavy documentation.

Their popularity comes with problems though: large numbers of ipynb files accumulate in repos, many of which are in a broken state. As a result, it is difficult for people to re-run, or even understand your notebooks.

This tutorial describes how you can use the pytest plugin nbmake to automate end-to-end testing of notebooks.

Pre-requisites

This guide builds on fundamental skills in testing Python projects which are described in Python Continuous Integration and Deployment From Scratch.

Before proceeding, please ensure you have covered these basics and have your Python 3 toolchain of choice (such as pip + virtualenv) installed in your development environment.

Demo Application

It’s common for Python projects to contain a directory of notebook files (known by the .ipynb extension) which may contain any of the following contents:

proof-of-concept modeling code
example API usage docs
Lengthy scientific tutorials

For the purpose of this tutorial, we are going to learn how to automate simple end-to-end tests on some notebooks containing Python exercises. Thanks to pytudes for providing this example material.

Fork and clone the example project on GitHub.

Inside this repo, you will find a directory ipynb containing notebooks. Install the dependencies in the requirements.txt file, optionally creating yourself a virtual environment first.

Before proceeding to the next step, see if you can run these notebooks in your editor of choice.

Testing Notebooks Locally

It’s likely that up to this point, your test process involves you manually running notebooks through the Jupyter Lab UI, or a similar client. This is both time-consuming and error-prone though.

Let’s start automating this process in your development environment using nbmake as an initial step forward.

Nbmake is a python package that serves as a pytest plugin for testing notebooks. It is developed by the author of this guide and is used by well-known scientific organizations such as Dask, Quansight, and Kitware. You can install it using pip, or a package manager of your choice.

pip install nbmake==0.5

If you did not already have pytest installed, it will be installed for you as a dependency of nbmake.

Before testing your notebooks for the first time, let’s check everything is set up correctly by instructing pytest to simply collect (but not run) all notebook test cases.

➜ pytest --collect-only --nbmake "./ipynb" 
================================ test session starts =================================
platform darwin -- Python 3.8.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/a/git/alex-treebeard/semaphore-demo-python-jupyter-notebooks
plugins: nbmake-0.5
collected 3 items                                                                    

<NotebookFile ipynb/Boggle.ipynb>
  <NotebookItem >
<NotebookFile ipynb/Cheryl-and-Eve.ipynb>
  <NotebookItem >
<NotebookFile ipynb/Differentiation.ipynb>
  <NotebookItem >

============================= 3 tests collected in 0.01s =============================

As you can see, pytest has collected some notebook items using the nbmake plugin.

Note: If you receive the message unrecognized arguments: --nbmake then the nbmake plugin is not installed. This may happen if your CLI is invoking a pytest binary outside of your current virtual environment. Check where your pytest binary is located with which pytest to confirm this.

Now that we have validated that nbmake and pytest are working together and can see your notebooks, let’s run them for real.

➜ pytest --nbmake "./ipynb"
================================ test session starts =================================
platform darwin -- Python 3.8.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/a/git/alex-treebeard/semaphore-demo-python-jupyter-notebooks
plugins: nbmake-0.5
collected 3 items                                                                    

ipynb/Boggle.ipynb .                                                           [ 33%]
ipynb/Cheryl-and-Eve.ipynb .                                                   [ 66%]
ipynb/Differentiation.ipynb .                                                  [100%]

================================= 3 passed in 37.65s =================================

Great, they have passed. These are simple notebooks in a demo project though. It is very unlikely all of your notebooks will pass the first time in a few seconds, so let’s go through some approaches for getting complex projects running.

Speed up Execution with pytest-xdist

Large projects may have many notebooks, each taking a long time to install packages, pull data from the network, and perform analyses.

If your tests are taking more than a few seconds, it’s worth checking how parallelizing the execution affects the runtime. We can do this with another pytest plugin pytest-xdist as follows:

First, install the xdist package. It is a pytest plugin similar to nbmake and will add new command-line options.

pip install pytest-xdist

Run with the number of worker processes set to auto using the following command

pytest --nbmake -n=auto "./ipynb"

Ignore Expected Errors in a Notebook

Not all notebooks will be easy to test automatically. Some will contain cells that need user input or will raise uncaught exceptions to illustrate some functionality.

Fortunately, we can put directives in our notebook metadata to tell nbmake to ignore and continue after errors are thrown. You may not have used notebook metadata before, so it's a good time to mention that notebooks are just JSON files, despite their ipynb extension. You can add custom fields to extend their behavior.

Your development environment may provide a UI for adding metadata to your notebook file, but let's do it in a raw text editor for educational purposes:

Open your .ipynb file in a simple text editor such as Sublime.
Locate your kernelspec field inside the notebook's metadata.
Add an execution object as a sibling to the kernelspec

This should leave you with something like this:

{
  "cells": [ ... ],
  "metadata": {
    "kernelspec": { ... },
    "execution": {
      "allow_errors": true,
      "timeout": 300
    }
  }
}

Now you can re-run the notebook to check that the JSON is valid and errors are ignored.

Write Executed Notebooks Back to the Repo

By default, tests are read-only: your ipynb file remains unchanged on disk. This is usually a good default because your editor may be open whilst you are running.

In some situations however, you may want to persist the executed notebooks to disk, if you need to

Debug failing notebooks by viewing outputs
Create a commit to your repository with notebooks in a reproducible state
Build executed notebooks into a doc site, using nbsphinx or jupyter book

We can direct nbmake to persist the executed notebooks to disk using the overwrite flag:

pytest --nbmake --overwrite "./ipynb"

Exclude Certain Notebooks from Test Runs

Some notebooks may be difficult to test automatically, due to requiring user inputs via stdin or taking a long time to run.

Use pytest’s ignore flag to de-select them.

 pytest --nbmake docs --overwrite --ignore=docs/landing-page.ipynb

Note: this will not work if you are selecting all notebooks using a glob pattern such as ("*ipynb") which manually overrides pytest's ignore flags.

Automate Notebook Tests on Semaphore CI

Using the techniques above, it is possible to create an automated testing process using Semaphore continuous integration (CI). Pragmatism is key: finding a way to test most of your notebooks and ignoring difficult ones is a good first step to improving quality.

Start by creating a project for the repo containing your notebooks. Select "Choose repository".

Next, connect Semaphore to the repo containing your notebooks.

Skip past "Add People" for now, so we can set up our workflow from scratch (even if you are using the demo repo).

Semaphore gives us some starter configuration, we are going to customize it first before running it.

Create a simple single-block workflow with the following details:

For Name of the Block we will use Test
For Name of the Job we will use Test Notebooks

For Commands we will use the following:

checkout
cache restore
pip install -r requirements.txt
pip install nbmake pytest-xdist
cache store
pytest --nbmake -n=auto ./ipynb

For Prologue use the following to configure Python 3:
```
sem-version python 3.8
```

Now we can run the workflow. If the workflow fails, don't worry, we'll address common problems next.

Troubleshooting Some Common Errors

Add Missing Jupyter Kernel to Your CI Environment

If you are using a kernel name other than the default ‘python3’. You will see an error message when executing your notebooks in a fresh CI environment: Error - No such kernel: 'mycustomkernel'

Use ipykernel to install your custom kernel if you are using Python.

python -m ipykernel install --user --name mycustomkernel

If you are using another language such as c++ in your notebooks, you may have a different process for installing your kernel.

Add Missing Secrets to Your CI Environment

In some cases, your notebook will fetch data from APIs requiring an API token or authenticated CLI. Anything that works on your development environment should work on Semaphore, so first check this post to see how to set up secrets.

Once you have installed secrets into Semaphore, you may need to configure the notebook to read the secrets from an environment variable available in CI environments.

Add Missing Dependencies to Your CI Environment

The python data science stack often requires native libraries to be installed. If you are using conda, it is likely that they will be covered in your normal installation process. It is slightly less likely if you are using standard python libraries.

If you are struggling to install the libraries that you need, have a look at creating a CI docker image. It will be easier to test locally and more stable over time than using Semaphore’s default environments.

Please remember the advice on pragmatism; you can often achieve 90% of the value in 10% of the time. This may involve tradeoffs like ignoring notebooks that run ML models requiring a GPU.

Conclusion

This guide demonstrated how you can automate testing of Jupyter Notebooks so you can maintain their reproducibility.

Notebooks have proven to be an effective tool for constructing technical narratives. Due to their novelty however, some software teams rule out their usage until clearer processes emerge to test, review, and reuse their contents.

There is still some way to go before notebook technologies fit into software projects seamlessly but now is a good time to get a head start. You may want to investigate some of the following further work items to further improve your operations:

Use pre-commit and nbstripout to remove bulky notebook output data before committing changes
Use jupyter book to compile your notebooks into a beautiful documentation site
Use ReviewNB to review notebooks in pull requests.

Thanks for reading, and please share any tips for working with notebooks with us!

This article was contributed by Alex Remedios.

Automated Testing: The Cornerstone of CI/CD

Marko Anastasov — Wed, 28 Apr 2021 11:10:56 +0000

This post was contributed by Ferdinando Santacroce.

In the Italian language, there is a single word for computer science: Informatica. It is a portmanteau of “informazione automatica” and, as you may have already spotted, it means something like information automation.

Automation is a pillar of information technology. Among experienced software professionals, it becomes instinctual. Every time you find yourself doing the same job over and over again, the desire to write a script and automate a repetitive task asserts itself.

This attitude is key for software developers, but sometimes things get overcomplicated. Then, automation becomes difficult, and manual work becomes necessary. This article describes what automated testing is, the path to a fully automated test suite, and the challenges that lie along the way.

What is automated testing?

Like any other material good, software needs to be tested before reaching the customer. The obvious way to test something is to use it for a while to make sure it behaves as expected. This is the way most developers in the past used to check their work before shipping it.

After all, you have written it, you have it on your computer, you know how to run it, and you know how to quickly test it as you make changes.

The flip side is that you as the creator are biased. You might forget to test some parts or might not realize the ramifications of a change. Also, you can’t be 100% sure that your code will work in an environment that isn’t your development machine.

Even if you do everything right, testing is a boring task — a time-consuming activity where skilled, creative professionals are forced to do mindless drudgery. Historically, the countermeasure was hiring people to test, making developers happy and removing biases.

This seemingly win-win solution makes things worse more often than not. Developers lose the overall picture and awareness of all the moving parts when they aren’t involved in testing. Testers, on the other hand, spend an enormous amount of time just making things work and have to bother developers whenever they find something they don’t understand.

The role of a QA team when testing is automated

Testers are part of what is known as the Quality Assurance (QA) team. They are paramount for shipping high-quality products, so it’s important to let them do their job efficiently. They do not have to test everything every time, nor catch unhandled exceptions or 404 errors. They are there to help developers and companies to deliver good quality software. Otherwise, the “QA is doing the testing anyway” mindset settles in, giving rise to dysfunctional behaviors.

The inflection point for automated testing

In 1999, the book Extreme Programming Explained laid the groundwork for a revolution, which later resulted in the Manifesto for Agile Software Development. In this text, we find the cornerstones of modern software development:

Rapid feedback loops.
Simple and safe ways to modify code. As Kent Beck said, “make the change easy, then make the easy change.”
A happy and satisfied team when the work is done.

In this field, automated testing plays a key role. Let’s have a look at some kinds of tests we can put into use.

From unit tests to end-to-end tests

The topic of testing is very broad and there are dozens of different types of tests, depending on the needs of the developer. Let’s suppose, for example, that we want to create software that can manage prices in a supermarket. The code below is an example of a unit test:

@Test
 public void When_I_add_an_apple_the_system_charges_50_cents() {
 CashRegister register = new CashRegister();
 register.add("apple");

 int expectedCost = 50;
 int actualCost = register.getAmount();

 assertEquals(expectedCost, actualCost);
}

In this test, we are verifying that our cash register charges 50 cents for an apple. Although simplified, the example gives a good idea of what it means to test a “unit of code” (the cash register), in a very focused way. If the price of an apple changes or the system stops pricing apples as expected, this test will alert us. A unit test validates an assertion on a specific portion of code.

There are other types of tests, aimed at ensuring the expected behavior of the application as a whole, from the graphical interface to the database. These kinds of tests are called acceptance and end-to-end (E2E) tests.

Our range of tests, therefore, goes from unit tests, which are focused on the technical details, to acceptance tests, which instead show that the required business objectives are being met.

Feature: Supermarket Bundles        
           To increase the revenues as a Supermarket Manager, 
           I want to apply discounts and offers to my customers

Scenario: if customer buys 3 apples, he pays $1.00
    Given the apples costs 50 cents each
    Given the “buy 3 apples, pay 2” promotion 
    When the cashier scans 3 apples
    Then the cash register charges $1.00

In the excerpt above, we see an example of a specification written in Gherkin, a language made famous by Cucumber and used in Behavior Driven Development (BDD). Despite appearing as natural language, there are automated tests hidden below the statements in this scenario.

The characteristics of a good test

All good tests have some features in common.

A good test is deterministic

A good test is deterministic and idempotent. It doesn’t matter where and when it runs, it should always produce the same (verifiable) outputs given the same inputs. Nothing is more frustrating than an unreliable test that fails randomly due to unknown or uncontrollable conditions.

Writing deterministic tests is not always easy, especially if the codebase was not designed to be testable from the beginning. Good programmers should take all the time they need to write deterministic tests.

A good test is fully-automated

A good test must be endlessly and effortlessly repeatable. As a developer works to implement new features, an external server can work to ensure the expected behavior. This is achieved by continuously running all the tests without any additional effort.

Machines are good at repetitive tasks, so let them do what they were designed for. Leave the developers with the higher and more rewarding task of satisfying users by unleashing their ingenuity.

A good test is responsive

A good test provides quick, honest feedback. Experienced developers know that testing is primarily about feedback. Having a test suite that offers feedback after hours or days is a big problem because it breaks cognitive flow. A developer can more quickly and effectively fix bugs the sooner they are discovered.

Build a robust test suite

A solid test suite has tests separated according to how long they take, their complexity, and the resources available. Unit tests run very fast — usually a matter of milliseconds — while end-to-end tests are slower, and more often than not they require staging environments to be properly installed.

In the end, a good developer should build a well-organized test suite over time, and write plenty of unit tests to run continuously. Here’s where a CI/CD pipeline comes into play.

The heart of Continuous Integration (CI)

At regular intervals, developers integrate changes with those of the other team members, using a versioning tool such as Git. A well-established team has a strong Continuous Integration policy and a CI pipeline that, upon noticing the changes, will take the new code from the repository and run tests on it — from quick unit tests to more rigorous acceptance tests. More costly tests can be run less frequently, such as once or twice a day. This lets the developers know immediately, with a good degree of confidence, if something stops working.

As in many aspects of life, there are trade-offs in testing strategy. It’s up to the developers and testers to negotiate the best compromise, ensuring quality and efficiency.

Simplicity and automation are prerequisites for safety

As mentioned earlier, software professionals must ensure the quality of their own work. Quality means low defectiveness, and low defectiveness means a vast amount of testing. Thus, writing good tests and building a system that can execute them quickly and accurately isn’t optional. Teams must commit to building a safety net with CI/CD to ensure smooth development.

The Myths of Test Automation

Despite the growing awareness achieved by companies and software developers in recent years, test automation is still plagued by some myths.

Testing slows down development

Those who claim that testing slows down development are both wrong and right. They’re right because in some cases, especially in TDD, writing tests makes developers take the time to think ahead about the result they want to achieve, setting aside the frenzy of writing code for a moment. In this case, slowing down is an act of will, a precise means of focusing on the problem ahead.

In the long run, however, those who think that writing tests are a waste of time are wrong. We must think of the time spent writing tests as an investment because a failure-safe environment makes it possible to make changes and experiments at a very high speed. Developers can come to a solution in a fraction of the time it would take if they constantly had to be careful about where they put their feet.

It’s like walking home on the sidewalk versus walking on a tightrope stretched between two buildings: tightrope walkers are not famous for their speed.

Abraham Lincoln used to say “Give me six hours to chop down a tree and I will spend the first four sharpening the axe“. Testing automation is like having a little helper that constantly sharpens your axe, so you can “hack” down any problems that pop up without spending hours sharpening it yourself first.

Testing is only for finding bugs

Another myth is that testing is only for finding bugs. Going after bugs is one possible purpose, but certainly not the only one. The real advantage lies in making bugs easily detectable and fixable after every single change. That is what test automation is all about: making it easier for a developer to alter code because they know they have a safety net.

You must achieve 100% coverage

Modern development environments allow you to calculate a metric called code coverage. It represents the amount of code covered by tests. It is quite natural for the non-test-savvy to think that 100% coverage guarantees the total absence of bugs in their codebase. This belief, apart from being incorrect, is a harbinger of bad practices.

Coverage only guarantees that there is at least one test that puts a certain portion of code under scrutiny. It does not guarantee that said test makes sense, that it verifies all the possible combinations of inputs, or that it will always behave as expected.

Writing tests is a profession somewhere between art and science, and coverage is certainly not the tool that guarantees the quality of a test suite. Furthermore, the drive for 100% coverage leads developers to test trivial portions of code, wasting time and money, and feeding an unhealthy perception of procedural infallibility. Setting a more reasonable threshold, such as 80%, might be a good compromise.

How to define a good testing strategy?

At this point, the question arises: what are the steps to creating a good test suite? Let’s try to define a roadmap for a company or a team that wants to change course and start testing its codebase.

Start testing

No, this is not a joke: the first thing you need to do is start testing. Usually a simple query on a search engine like “ testing tools” provides plenty of hints and tutorials to get started.

But even if you’re using outdated or abstruse languages, you can always find a way to test your product. If there are no frameworks or ready-made tools on the market, you can always write tests using pure programming languages or with bare shell scripts.

To paraphrase Walt Disney, “If you can run it, you can test it.“

Identify a meaningful part, and write an automated test

In codebases that have developed with no inclination to testability, the road is uphill at first.

The first tests written should offer a good balance between value and affordability, i.e. code that is sufficiently easy to approach, but that at the same time can provide you with some value. Testing a part that never breaks, a trivial feature, or one that almost nobody uses is not helpful. Moreover, going straight to the main feature of your product usually ends with brittle tests and frustrated developers.

Most of the time, we can start by writing some end-to-end tests that try to emulate the behavior of the user. If we have a web application, for example, we can begin by training a browser with tools like Selenium or Cypress. Let’s not, however, limit ourselves to tools; we also have to be ready to manually write our test routines, shell scripts, and whatever else is needed to make the test automatic.

In the journey we take, we always discover useful details for increasing our understanding of the codebase, details that will be valuable when we go to modify or refactor bits of code. Once we understand the basic principles of testing, we can start learning about testing frameworks, mocking libraries, and all the tools that make a developer’s job easier.

Create your safety net

A good test suite is a safety net. In the beginning, when you don’t have tests, every change is an unknown risk. Over time, and with the addition of tests, the risk decreases, confidence increases, and development accelerates.

It’s a long process. You can’t hope to solve all the accumulated problems in a few weeks. The important thing is to keep adding tests, and reinforcing testing practices in the team.

Make automation an integral part from the beginning. If a new project is about to take off, don’t waste the opportunity to start off with the right foot with a well-rounded testing strategy and a solid set of CI tools. Once the right procedure for conducting a test has been identified, the test must be automated, and then it must be run in a secure, isolated environment: a CI/CD pipeline.

This is about feedback: the longer you wait, the later you will know if your automated tests are as isolated and idempotent as you want them to be, or if they “just work on your computer”.

Monorepo Support Is Now Generally Available on Semaphore

Marko Anastasov — Wed, 07 Apr 2021 16:18:47 +0000

Our mission at Semaphore is to empower engineers to ship great products by providing them with a state-of-the-art CI/CD experience. We’re excited to announce that Semaphore now provides out-of-the-box support for monorepo projects.

What is monorepo? 🚝

A monorepo is a version-controlled code repository that holds many projects. While these may be related, they are often logically independent and run by different teams.

What is monorepo? (and should you use it)

Tomas Fernandez for Semaphore ・ Mar 31 '21

How it works? 🤔

The key building block of monorepo CI/CD pipelines on Semaphore is the change_in function which detects if a file or set of files have changed in a particular Git commit range.

You can use the change_in function to:

Run custom workflows for changes on specific files. Use change_in in combination with glob patterns to define jobs that will run only when certain files change.
Skip unnecessary checks. No need to run that whole test suite when you change README.md. Save some CI time by using exclude option to define files and folders you want to ignore.
Continuously deploy the right things. Combine change_in with automatic promotions to deploy what has changed.

So what? 😜

Skipping parts of your code which were not affected by a change can lead to a dramatically faster feedback loop.

For example, the front-end engineering team at BlueLabs was able to reduce their build time from 17mins to 4mins.

How can I use this? 😄

Semaphore is free for personal, hobby and open source projects, so giving this a spin is pretty easy.

You can fork a pre-made monorepo demo project and run it yourself.

Or you can just add your own project and play with it.

Happy building!

A more PR-ish version of this announcement was originally published on the Semaphore blog.

Talking CI/CD on Changelog’s Go Time Podcast

Marko Anastasov — Wed, 10 Feb 2021 14:49:25 +0000

We're talkin' CI/CD

Go Time

Your browser does not support the audio element.

1x initializing... ×

I had an honor to speak on Changelog’s Go Time podcast along with Jérôme Petazzoni on Continuous Integration and Delivery (CI/CD). We discussed:

What problems does CI/CD solve?
What does it look like when it’s done well?
What are the pitfalls in doing CI/CD and how to avoid them?

🎙 Listen and/or read the transcript here — I hope you enjoy the show!

Android Continuous Integration and Deployment Tutorial

Marko Anastasov — Tue, 10 Nov 2020 13:25:42 +0000

As we work on an Android app, we test our code regularly — hopefully with the help of automated tests. And when it's time to release a new version, we assemble the source code into an APK or AppBundle format which we deploy into the Google Play Store.

It seems like a simple process, but there are several other steps involved, like signing the package and updating the version code and name.

After a while, going through this process manually gets very tedious and tiring, so how can we automate this process?

Enter Semaphore

Semaphore is the fastest continuous integration (CI) and continuous delivery (CD) service which helps you automate your workflow by building, testing, and deploying your Android project. This way you can focus on what matters: your code.

In this tutorial we will explain how to set up Semaphore for a native Android project.

Prerequisites

If you wish to follow this tutorial you'll need the following:

Git and a GitHub account.
A Semaphore account. You can get one for free at semaphoreci.com.
A Google Play Developer account to publish the app.

How to finish this tutorial in 1 minute

To get started really quickly, do this:

In Semaphore, follow the link in the top navigation to create a new project.

From the examples list, select Android (Kotlin).

Semaphore will fork a demo project:

semaphoreci-demos / semaphore-demo-android

And set up a complete CI/CD pipeline for you in less than a minute:

That’s it. You’re done. *mic drop*

The demo is a simple app that lists all the open source Semaphore demo projects on GitHub. It includes unit and integration tests, and has the Gradle Play Publisher plugin set up that adds a Gradle task to publish your app to the store.

You can see this app live on the Play Store.

However, I know that you’re most likely here to learn how to set up continuous integration for your own Android project. So let's see how to do that using the demo project as a foundation.

Adapting these instructions to your project should not be too difficult. But if you do run into a challenge, feel free to ask a question in the comments section. 💬

Configuring CI/CD on Semaphore from scratch

After creating your Semaphore account you'll need to create your first Semaphore project. Your Semaphore configuration is stored in a directory at the root of your repository called .semaphore. The entry point of your CI/CD pipeline is defined by the semaphore.yml file.

In this tutorial we will explain how to configure an Android CI/CD workflow by reading the configuration files. That way you’ll learn the most about Semaphore and Android continuous integration in general.

But you don't have to write YAML by hand. You can also use Semaphore’s graphical workflow builder for a more visual approach.

👉 See this public workflow to get a taste of Semaphore’s UI and the final project running.

Continuous integration: the safety net for your Android project

We want to perform regular checks on the code by running a linting tool and by running the project tests, and we want those to be automated. Let's check on what can we do on Semaphore and edit the semaphore.yml configuration file.

👉 You can see the full file of the demo project on GitHub.

Agent: where your code runs

agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
  containers:
    - name: main
      image: registry.semaphoreci.com/android:29

Here we're defining the environment where the build will run. We want a Linux machine with 2 vCPUs, 4 GB of RAM that runs Ubuntu 18.04 OS. We also want to run our jobs in a container that has the latest (at the time of writing) Android SDK installed, provided by Semaphore Container Registry.

Semaphore always caches the two latest stable versions of Android SDK to ensure that the machine is ready in no time, making our builds faster.

Global job configuration

global_job_config:
  env_vars:
    - name: ADB_INSTALL_TIMEOUT
      value: "10"
  secrets:
    - name: android_keys
  prologue:
    commands:
      - checkout
      - mv ~/release-keystore.jks ~/$SEMAPHORE_GIT_DIR/app/release-keystore.jks
      - cache restore gradle-wrapper
      - cache restore gradle-cache
      - cache restore android-build

Semaphore's Global Job Config allows us to define a set of configurations that are shared across the whole pipeline. This is very helpful since it allows us to define everything in one
place instead of repeating it in every task.

Here we're defining a new environment variable ADB_INSTALL_TIMEOUT so that
adb doesn't timeout while we're setting up the emulator and running the
integration tests.

We're also saying that we want to use the global secrets android_keys that will allow us to use our keys and passwords defined as secrets, as environment variables in the build.

Specifically on the demo project we're saving the following secrets on android_keys:

release-keystore.jks: the keystore file used to sign our release builds;
RELEASE_KEYSTORE_PASSWORD: our keystore password;
RELEASE_KEY_ALIAS: our keystore key alias;
RELEASE_KEY_PASSWORD: our keystore key password;
service-account-key.json: the service account key, used to deploy the demo app to the Play Store.

This is the ideal place so save sensitive data, such as keys. Be sure to use secrets on your builds instead of pushing sensitive information into your git repository.

The prologue section allow us to run a set of commands before the job begins. We're checking out the code from the git repository, moving our keystore file to the correct path and restoring our cache so builds are faster.

Pipeline blocks

Here's where we can add all the build tasks we want to run. We define the jobs
of each task, and the commands we want to execute in this pipeline. Let's start
at the top.

Build Block

- name: 'Build'
  task:
    jobs:
      - name: 'Build Project'
        commands:
          - ./gradlew bundle
    epilogue:
      on_pass:
        commands:
          - cache clear
          - cache store gradle-wrapper ~/.gradle/wrapper
          - cache store gradle-cache ~/.gradle/caches
          - cache store android-build ~/.android/build-cache

The first thing we want to do is to build our project by running ./gradlew bundle to check that everything is ok. Then, in the epilogue, that will only run if this job is successful, we are refreshing the cache.

Verification Block

- name: 'Verification'
  skip:
    when: "pull_request !~ '.*'"
  task:
    jobs:
      - name: 'Analyze Code'
        commands:
          - ./gradlew lint
      - name: 'Unit Tests'
        commands:
          - ./gradlew test
      - name: 'Integration Tests'
        commands:
          # Install the required tools and the emulator itself
          - sdkmanager "platform-tools" "platforms;android-29" "build-tools;30.0.0" "emulator"
          # Install system images for the emulator
          - sdkmanager "system-images;android-29;google_apis;x86"
          # Create an emulator with the installed system images
          - echo no | avdmanager create avd -n test-emulator -k "system-images;android-29;google_apis;x86"
          # Start the emulator with no audio, boot animation, window, and with GPU acceleration off
          - emulator -avd test-emulator -noaudio -no-boot-anim -gpu off -no-window &
          # Wait for the emulator to boot completely
          - adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;'
          # Dismiss the emulator lock screen and wait 1 second for it to settle
          - adb shell wm dismiss-keyguard
          - sleep 1
          # Disable window and transition animations. This is required to run UI tests correctly
          - adb shell settings put global window_animation_scale 0
          - adb shell settings put global transition_animation_scale 0
          - adb shell settings put global animator_duration_scale 0
          - ./gradlew connectedAndroidTest
    epilogue:
      always:
        commands:
          - artifact push job --expire-in 2w --destination reports/ app/build/reports/

Now the real work begins. This is the verification block where we're doing a
lint check and running our tests. We also have a skip condition that tells Semaphore that we only want to run this block when a pull request is
open.

Here's what's happening:

./gradlew lint runs the Android Lint that analyzes the code to check if there's any problems.
./gradlew test will run all the Unit Tests on our project.
./gradlew connectedAndroidTest will run the Integration Tests, after setting up the emulator.

Usually, this is enough to verify the code and should always be checked/ran before merging and deploying a new app version so that we're sure that we're not introducing bugs or bad code into the project. You can add more jobs here if you need to perform additional checks.

After everything completes, on the epilogue, we're uploading our test reports as artifacts so if something goes wrong with our tests we can consult the reports to see more details, making our lives a bit easier.

👉 It's important to note that every thing will run in parallel making Semaphore builds extremely fast.

Promotions

Promotions are used when we want to branch out our pipeline when certain conditions are met. This is perfect for deployments.

promotions:
  - name: Internal Deploy
    pipeline_file: deploy-internal.yml
    auto_promote:
      when: "result = 'passed' AND branch = 'dev'"
  - name: Beta Deploy
    pipeline_file: deploy-beta.yml
    auto_promote:
      when: "result = 'passed' AND branch = 'master'"

We have two other pipelines, and like so two promotions. We want to promote to the pipeline that deploys our app to the internal track when we push new code to dev, and to the beta track when we push new code to master.

Here's the pipeline in it's full glory. Notice how quick it is.

Continuous Deployment: you can do it with Android too!

After verifying that everything is ok with our project we want to automate deploys.

Check the full configurations for the internal pipeline and beta pipeline.

Most of the configuration is the same as explained before. We only added a command to the global config to move the service account key to the correct path as that is needed to deploy the app and added a new block.

Deployment Block

- name: 'Internal Deployment'
  task:
    jobs:
      - name: "Deploy to Play Store - Beta Track"
        commands:
          - ./gradlew publishReleaseBundle --track beta

Here we're simply running the publish task that was created by the Google Play Publisher gradle plugin. The only thing that differs from the internal pipeline is the argument track that specifies that we want to deploy to the internal track instead. Also, we don't have to worry about incrementing the versionCode every time because we're using the Semaphore workflow number environment variable for that, that is always incremented when new a new build is running.

On to you

Remember to check the complete demo project on GitHub for a complete example and be sure to explore the code on your own. Feel free to fork it and use it as the base of your next Semaphore project.

There are still somethings that you can do to improve the workflow, like adding Slack notifications, using webhooks, updating a CHANGELOG file or even creating GitHub releases, but the nice thing about Semaphore, and CI/CD tools in general, is that now you can finally focus on what matters: the code for your next Android app.

Have questions about this tutorial? Want to show off your results? Leave a comment below. 🙇‍♂️

Talking CI/CD and containers on the Docker Live Show

Marko Anastasov — Tue, 13 Oct 2020 10:24:57 +0000

I'd like to share with you a video of me being a guest at Bret Fisher's Docker Live show which I think was a great discussion.

The questions were pouring in from the start so the entire episode had a great flow.

Some of the things we've discussed:

The origins of Semaphore
Git & GitHub flow
The CI/CD with Docker and Kubernetes book
The importance of automated tests and how to get started
How to set up and optimize CI/CD pipelines when using Docker
Semaphore's Workflow Builder as a way of making CI/CD more accessible to developers
...and more!

Watch it — and let me know if you have any questions or feedback:

When Does It Make Sense to Use Docker?

Marko Anastasov — Wed, 30 Sep 2020 14:46:14 +0000

Using Docker / containers makes sense only if you need to standardize deployment pipelines. Otherwise it’s a waste of time.

When do you need to standardize anything? When you're dealing with a a large quantity.

So, either you:

Have many (as in, double digits) microservices comprising one system, or
Are in a very large organization with many development teams.

So if you have one or two apps running on Heroku / Beanstalk — leave them there. You’re not missing out on anything. It's pure overhead.