DEV Community: Doug Sillars

Database branching in Django apps using GitHub actions

Doug Sillars — Mon, 02 Dec 2024 18:36:27 +0000

Creating online previews of your applications is a great way to test that all the required functionality is present. When building and testing a dev build of your application from a pull request (PR), the last thing you want is for your tests to affect your production database. Using a test branch of the production database ensures that the production database remains untouched, ensuring no accidental deletion of data or adding test data into the production database.

In this post, we’ll create a set of GitHub Actions to automate the testing process of a pull request. Our GitHub Action will run when the pull request is created, generate a test branch of the production database, and deploy the code to Digital Ocean. Once the PR is merged, a second GitHub Action will rebuild the Digital Ocean app with production code (and database), and the test database branch will be deleted.

By finishing this article, you’ll be able to automate the creation of app previews using NeonDB branches, Digital Ocean, and Django. Let’s jump right in!

Setup

Code repo

To begin, we’ll use the Django Neon q uickstart repo on GitHub. Create a fork and clone your fork locally. Follow the setup instructions in the README, and run the application locally to ensure it is up and running. (You’ll need a NeonDB account to add the environmental variables that connect to NeonDB and power the app.)

Digital Ocean

Digital Ocean has a simplified process for launching applications on its platform. You can be up and running in minutes with just a few configuration steps.

You’ll need an account at Digital Ocean to deploy your application. From the Digital Ocean dashboard, select “Create” → App Platform. Connect to GitHub, and choose the django-neon-quickstart, branch main. Click next. I ran this on the $5/month instance. In the “build phase,” click edit and add the three commands below:

pip install -r requirements.txt
python manage.py makemigrations
python manage.py migrate

These are the steps you ran to get the repo running locally—we’re just repeating it on Digital Ocean.

Update the “Run” command to use gunicorn:

gunicorn django_neon.wsgi:application --bind 0.0.0.0:8000

Finally, update the HTTP Port to 8000 to match the port in the repository.

In step two of the setup process, update the environmental variables. You can do a bulk upload and copy and paste in your .env from the local repo as shown below:

Click through the rest of the commands, and on completion, the app will deploy and be available for use on the internet.

Note that in this repository the deployed code is the main branch. We would like to display the PR preview from the dev branch. (Change the names main and dev to whatever branches you wish to preview.) Let’s continue our setup.

GitHub Secrets

We need to add three GitHub secrets to the repository. Add secrets by clicking “Settings” on the top menu. Then: “Secrets and Variables” → Actions, and add repository secrets.

Here are the three secrets to be added:

DO_KEY: A key from Digital Ocean with scopes to create, read, update, and delete apps. You can create this from the Digital Ocean dashboard under API.
NEON_API_KEY: Create your Neon API key at the NEON dashboard. Click your Avatar in the upper right corner, select Account Settings, and then choose API keys to create an API key.
NEON_PW: This is the PGPASSWORD from the .env file.

Digital Ocean app spec files

The App Spec file defines how the build process will be run at Digital Ocean. You can find your App Spec file in your App’s dashboard under “Settings.” (It will be autogenerated when you create your project.)

In your GitHub Repository, create a .do directory, and make two copies of the App Spec file: app.yaml and default.yaml. These will be used by our GitHub Actions to edit the Digital Ocean Application.

app.yaml is what will be provisioned on a pull request being opened:

    alerts:
    - rule: DEPLOYMENT_FAILED
    - rule: DOMAIN_FAILED
    features:
    - buildpack-stack=ubuntu-22
    ingress:
      rules:
      - component:
          name: django-neon-quickstart
        match:
          path:
            prefix: /
    name: **seal-app-dev**
    region: nyc
    services:
    - build_command: |-
        pip install -r requirements.txt
        python manage.py makemigrations
        python manage.py migrate
      environment_slug: python
      envs:
      - key: PGHOST
        scope: RUN_AND_BUILD_TIME
        value: **new_host**
      - key: PGDATABASE
        scope: RUN_AND_BUILD_TIME
        value: neondb
      - key: PGUSER
        scope: RUN_AND_BUILD_TIME
        value: neondb_owner
      - key: PGPASSWORD
        scope: RUN_AND_BUILD_TIME
        value: **new_password**
      github:
        branch: **dev**
        deploy_on_push: true
        repo: dougsillars/django-neon-quickstart
      http_port: 8000
      instance_count: 1
      instance_size_slug: apps-s-1vcpu-1gb-fixed
      name: django-neon-quickstart
      run_command: gunicorn django_neon.wsgi:application --bind 0.0.0.0:8000
      source_dir: /

There are four changes made in this file from the original at Digital Ocean:

name: add “-dev” to the end of the app name.
PGHOST: Value should be new_host
PGPASSOWRD:value should be new_password
github:branch dev replaces main.

When our GitHub Action runs:

The name of the application will change in the Digital Ocean dashboard, allowing the dev to see that the current state is a dev state.
new_host and new_password will be programmatically updated with the values from our newly created NeonDB branch.
We want to deploy the dev branch of our code to Digital Ocean to see the changes.

default.yaml is used to revert the App Spec to production.

The default.yaml has two changes:

name: Add “-prod” to the end of the name.
PGPASSWORD:value: new_password replaces the password.

(We’re not showing the entire file here for space reasons.)

This will change the name in the Digital Ocean dashboard to show that prod is visible. The password will revert to the original password from the primary branch of the database, and since the branch is main, the build will be the main branch.

With these changes, we are now ready to begin implementing our two GitHub Actions: “Create NeonDB Branch” and “Destroy NeonDB Branch.”

Create NeonDB branch

When a pull request is made to the main branch, this GitHub Action will fire and do a number of steps:

Create a development branch of the NeonDB database.
- Grab the host and password of this new DB.
Check out the GitHub Code.
Use a sed command to replace new_host & new_password placeholders in the .do/app.yaml file with the variables extracted in step 1a.
Install the Digital Ocean CLI.
Update the App Spec with the new yaml file.
Initiate a Digital Ocean deployment.

    name: Create Neon Branch and deploy dev to DO
    run-name: Create a Neon Branch 🚀
    on:
      pull_request:
        types: [opened]
        branches:
          - main
    jobs:
      Create-Neon-Branch:
        runs-on: ubuntu-latest
        steps:
          - name: Verify NEON API Key presence
            run: |
              if [ -z "${{ secrets.NEON_API_KEY }}" ]; then
                echo "NEON_API_KEY is empty"
              else
                echo "NEON_API_KEY is set"
              fi
          - name: Create Neon Branch
            id: create-branch
            uses: neondatabase/create-branch-action@v5
            with:
              project_id: "orange-violet-68318343"
              # optional (defaults to your primary  branch)
              parent: "main" 
              # optional (defaults to neondb)
              database: "neondb"
              branch_name: "development"
              username: "neondb_owner"
              api_key: "${{ secrets.NEON_API_KEY }}"
          - run: echo db_url ${{ steps.create-branch.outputs.db_url }}
          - run: echo host ${{ steps.create-branch.outputs.host }}
          - run: echo branch_id ${{ steps.create-branch.outputs.branch_id }}
          - name: Checkout code
            uses: actions/checkout@v2
          - name: Replace variables in YAML
            run: |
              sed -i 's|new_host|'"${{ steps.create-branch.outputs.host }}"'|g' .do/app.yaml
              sed -i 's|new_password|'"${{ steps.create-branch.outputs.password }}"'|g' .do/app.yaml
          - name: Install doctl
            uses: digitalocean/action-doctl@v2
            with:
              token: ${{ secrets.DO_KEY }}
          - name: Set environment variables
            run: |
              doctl auth init -t ${{ secrets.DO_KEY }}
              # Update the app with the new specifications from neon
              #  use active project id from DO url
              doctl apps update 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9  --spec .do/app.yaml
              doctl apps create-deployment 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9

Hints on creating your action:

The NeonDB project_id is in the URL string when you load the project in the NeonDB dashboard.
The UUID for your Digital Ocean application is also found in the dashboard URL.

Save this workflow in /.github/workflows.

Delete NeonDB branch

Once the PR has been tested and approved, we want to destroy the NeonDB branch and revert the Digital Ocean deployment back to production.

The steps in this are:

Delete the NeonDB development branch.
Check out the code.
Update the default.yaml with our DB password from the GitHub Secrets.
Update the App Spec and deploy the application at Digital Ocean.

    name: Delete Neon Branch with GitHub Actions Demo
    run-name: Delete a Neon Branch 🚀
    on:
      pull_request:
        types: [closed]
        branches:
          - main
    jobs:
      delete-neon-branch:
        runs-on: ubuntu-latest
        steps:
          - name: Delete Neon branch
            uses: neondatabase/delete-branch-action@v3
            with:
              project_id: "orange-violet-68318343"
              branch: development
              api_key: ${{ secrets.NEON_API_KEY }}
          - name: Checkout code
            uses: actions/checkout@v2
          - name: Replace variables in YAML
            run: |
              sed -i 's|new_password|'"${{ secrets.NEON_PW }}"'|g' .do/default.yaml
          - name: Install doctl
            uses: digitalocean/action-doctl@v2
            with:
              token: ${{ secrets.DO_KEY }}
          - name: Set environment variables
            run: |
              doctl auth init -t ${{ secrets.DO_KEY }}
              # Update the app with the new specifications from neon
              #  use active project id from DO url
              doctl apps update 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9  --spec .do/default.yaml
              doctl apps create-deployment 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9

Okay, that is a lot of code. Don’t forget to update the DO UUIDs to match your deployment. Push this all to your repo so that we can see our automation in action.

Here is the production version of the application running on Digital Ocean. I added a few extra elements for fun. The screenshot shows the mouse hover color on Ne (Neon).

Now, let’s make some changes to the code and start a pull request.

Create a dev branch for your code. Let’s change the colors in line 15 of/elements/templates/elements_list.html:

<li hx-delete="element/{{ element.id }}" hx-target="body" class="relative flex flex-col text-center p-5 rounded-md bg-[#7846a8] transition-colors hover:bg-orange-500 text-[white]">

This should make the boxes purple, with orange hover and white text.

Push the dev branch and open a pull request.

When the pull request is created, the “Create a Neon Branch” GitHub Action is called. A branch of the NeonDB is created, and the dev code is deployed to Digital Ocean.

Refreshing the application, we see the colors have been updated:

More importantly, we can test all we want without worrying about the production database. Any changes made on the dev branch are in the development branch of NeonDB. As a part of our testing, we deleted a number of entries—only Neon is left:

Since we’re happy with the PR, we can approve and merge the changes. This fires up the second GitHub Action: deleting the NeonDB development branch and pushing the production build to Digital Ocean:

The new colors are in prod, and the prod database was untouched by our testing on the PR!

Conclusion

In this post, we used GitHub Actions to automate creating and deleting a NeonDB database for testing pull request builds on Digital Ocean. If you would like to look at the code, it is available on GitHub. You’ll just need to wire in your NeonDB and Digital Ocean credentials to get up and running.

Automating Developer Relations Metrics with Low Code RunBooks

Doug Sillars — Mon, 24 Apr 2023 22:05:02 +0000

My job at unSkript is to spread awareness and excitement around the DevOps tooling we have built. But, I am also expected to provide reporting on various metrics around developer awareness and usage of our product. In this post, I walk through how I have automated the data collection process, so that I can spend more time creating content and building awareness.

A bit of background: at unSkript, we are building automation tools to reduce toil. In the DevOps/SRE space, toil is defined as the manual and repetitive work that needs to be done to keep everything shipshape. If you ask me – collecting metrics from a bunch of different services (Github, Google Analytics, internal databases, Docker,….), and aggregating them in one place – that sounds like toil. So let’s automate that away, and then I no longer have to think about it (until I decide to write a blog post about it, of course.)

Collecting the Data

unSkript is a tool to help you build RunBooks. A RunBook is a collection of steps (we call them Actions) that complete a task. For DevOps teams, that could be auto-remediation of your load balancers, running health checks on a K8s cluster, or even monitoring your daily Cloud costs.

I want to use these actions to collect a bunch of different data points, and store them all in one place. There are a few different ways that I use unSkript to collect the information:

Built in Actions

unSkript comes with hundreds of built-in Actions – simply drag & drop into your RunBook, configure your credentials, and you are ready to go! When using built-in Actions – unSkript can be thought of as essentially “no-code” to set up. There are several built-in Actions in unSkript that are well suited to collecting the data that I want to collect: Daily Unique users from Google Analytics, and the Github star count.

NOTE: Github stars as a DevRel metric can be controversial (IMO- it is useful as an indicator metric) but feel free to leave a comment below with your thoughts.)

Database Queries

Many of our stats are collected from Segment, and stored in a database (and that database is awesome for in depth detailed analysis). But I want to keep all of my high level statistics and data in one table, so, I’ll use the PostgreSQL connector to extract the datapoints I’d like into my dataset:

These Actions are “low-code” in that once you drag & drop the action and make the connections, you still need to create a SQL query to grab the results.

REST API

There are still a few more data points that I’d like to pull out of other tools. We have a REST API connector that makes this easy: set up your credentials and the headers you need – and you can create a new Action that extracts your data via API. These are also “low-code” but do require some understanding of how to make API calls – in order to set up the credentials properly.

For example: Docker Hub publishes the number of times our Docker Image has been downloaded. We can collect this number Each day using the REST API Action – and adding the endpoint and headers to the Action:

Storing & reporting our data

Once we have collected all of the data, we can create a message and post it on Slack for the team to see:

The message that is sent to the channel is a Python f string with variables added.

This is a fun way to update the team on a daily basis…but we also want to chart this data over time. To accomplish this, we have a table in PostgreSQL for our stats, and we just make an INSERT using the prebuilt Postgres Action (again this is low-code as you must write the SQL INSERT command:

The data in Postgres feeds a Grafana dashboard – allowing the team access to the latest data from our metrics – and the best part of it all is that there is no daily toil required. And many folks on the team just go to the dashboard to get the data – the DevRel team is no longer a bottleneck!

Progressive enhancement

As time goes on, more questions about data will arise.

As an example, since the number of Actions and RunBooks in GitHub keeps increasing, I was recently asked “how many Actions do we have in GItHub today?”

The first few times you are asked a question like this, you can probably get away with waving your hands, and a ballpark figure… but after being asked a few times, I knew I needed a “real” answer. Reusing an existing GitHub Action, I was able to create a file in Github with the counts that I needed. By dragging a new Action into my RunBook, writing a few lines of Python code (and a small change on the Postgres insert), I was able to easily extend the current data collection to include more data.

Aside: We can also leverage these values to create custom badges for the Github readme, and on the website – so creating the data has been a double win!

Scheduling

Now that I have built a RunBook that collects all of data we need (so far…) -> I want to automate the execution of the RunBook. Using unSkript’s Scheduler, I have set my RunBook to run at midnight GMT every day.

Now, I have daily reports being created for the team, that require ZERO work on my part!

Summary

Collecting and aggregating statistics via automation frees the team to can focus on our “real” work: creating more tools and applications – and no longer spend significant time on metric collection. At the same time, everyone has visibility into the project – showing the value of the DevRel team, without impacting their workload.

How does your DevRel team collect usage data? If you’d like to give unSkript a try, check out our Free trial. Join our Slack Channel, and I would be happy to chat with you on strategies to build your RunBook to collect your analytics data. I’ll also be happy to share the skeleton of my RunBook to get you started!

Automating the GitHub Nudge

Doug Sillars — Tue, 04 Apr 2023 16:46:28 +0000

Git is a tool of Actions. Millions of times a day, users checkout, push, pull and merge submissions to their repositories. The scale is staggering:over 3.5 billion contributions were made on GitHub in 2022. That’s 227 million Pull Requests merged, and over 31 million issues closed. However, What about the PRs and the issues that fall through the cracks and are forgotten? Should they be just left, forgotten (like the Island of Misfit toys in Rudolf the RedNosed Reindeer?)

In this post, we introduce an automated RunBook that introduces the “Github nudge.” Merriam-Webster’s definition for a nudge is “to prod lightly : urge into action.” The GitHub nudge identifies issues and PRs that have been sitting a while, and “nudges” the assignee to take a look. By not letting the team forget that the issues exist – they are more likely to be acted upon!

We’ve defined a few Actions in unSkript’s RunBook architecture to help us along this path:

Stale Issues

When issues have been assigned to a team member, but no work is being done on them, the issue has probably “gotten lost.” Everyone has a lot to work on, and sometimes these issues just lose priority – or get superseded by other tasks. That does not mean that they should just be ignored – resolving the issue will improve the project.

In unSkript, there is an Action to find “stale” issues: that is issues that are over a certain age.

This Action takes 3 input parameters – the Github owner and repository and the threshold (in days) upon which you define an issue as stale. In the case of our repository https://github.com/unskript/Awesome-CloudOps-Automation, the owner is unskript, the repo is Awesome-CloudOps-Automation, and we set the stale threshold at 14 days. There’s no real science around that day -it felt like a good number for our team.

The response is an array of issues that surpass that threshold. With this data, we can do some quick examinations:

Issues without an assignee

Issues that have not been assigned to a resource are not going to be worked on – we all have a lot on our plate already! In the case of Awesome-CloudOps-Automation – these are all “good first issues” for those interested in contributing. I like to scan this list once a week for changes – and to ensure that issues that do need work are properly assigned.

Issues with an assignee

If an issue has been around for 2 weeks, and is not yet resolved – its good to check on them to see the status – and that action is being made:

For example – it appears that issue 346 is assigned to me, and I am overdue with an update (oops!):

{'assignee': NamedUser(login="dougsillars"),
   'issue_number': 346,
   'title': '[Action]: GitHub Comment on an issue'}

In this case, the PR was already merged, but not connected to the issue, so it could be closed (whew!).

Stale Pull Requests

We can do the same thing for Pull Requests:

A stale PR almost feels worse than a stale Issue. When the work is completed, and ready (perhaps nearly ready) to be integrated into the codebase – there is a immediate improvement to the software. When. PR just languishes, the code does not improve, and sometimes further improvements are blocked The Action above gives a list of all PRs that are over a threshold (again we use 14 days) for a given GitHub repository. For example:

{331: '3 cost optimization runbooks'}

This output does not provide anyone that I can ‘nudge’, but we have another Action that we can use:

To generate all of the reviewers to nudge, I first create a list of the PR numbers that I obtained from the first Action (the output was named “stalePull”:

oldPRs = []
for pr in stalePull[1]:
oldPRs.append(list(pr.keys())[0])
print(oldPRs)

The Get Pull Request Reviewer Action takes 3 inputs – owner, repository and PR number.

Using the iteration command in unSkript, we can apply the list of pull requests (oldPRs) to the pull_request_number variable:

This will call the Action once per pull request – giving a full output of each PR, and who is left to review.

331: ['jayasimha-raghavan-unskript', 'shloka-bhalgat-unskript']

With a little Python, we can turn this around to list every user, and the PRs they should look at.

U03QU1K184X': [267, 153]

You may notice that the username is different. This is in anticipation of the next step:

The actual “nudge”

Ok, so we know whose issues and PR reviews are overdue. How do we alert them? At unSkript, the internal team uses Slack. I have created a table that compares each Github username with their Slack ID (that’s the weird variable above).

With the slack ID, I can now send a message to the team channel:

If you are listed below, can you please review the Pull requests next to your name? They have been open for 14 days.

<@U01UG9DRR7D>, please review pull requests [267, 153].
<@U03QU1K184X>, please review pull requests [267, 153].

Adding the @ in front of the username creates an “at” in Slack, so these two users have just been effectively nudged to look at their PRs. They have now been nudged in Slack to go into GitHub and take a look at the work that is being forgotten.

Automating the Nudge!

Using the Enterprise version (or the free trial) of unSkript, you can schedule each RunBook. The RUnBook I created for Awesome-CloudOps-Automation runs every Wednesday morning – alerting the team that there are some issues and pull requests that have been left behind and should be resolved.

How do you keep your Github issues and Pull requests chugging along? Do you have suggestions on how you might improve this RunBook? We’d love to hear about it in the unSkript Slack channel. Interested in trying out Github nudges with your team? All of the Actions described above are in our Open source (Docker instructions are in the readme), and in our free Trial.

Cloud Costs: Charting Daily EC2 Usage and Cost

Doug Sillars — Wed, 22 Mar 2023 09:25:44 +0000

When you buy a t-shirt at the store, you (generally) pay the same price for the shirt if you are an XS or a 2XL. In the cloud, the size of the Virtual machine you select has VERY large implications in the cost. At AWS, launching a t2.nano instance costs $.0058 an hour – or 14 cents a day, while a t3.2xlarge instance is $.4628 – or $11.11 a day.

When sizing a new system, it is common to go a “bit larger” in size to ensure that the service performs well. This has worked well for AWS, as over 50% of their revenue comes from EC2 instances. In a recent post, we built an automated RunBook to examine our daily spend for each AWS product, and EC2 (the green line) is far and away our biggest cost center:

When looking at your highest cost center – you may wish to drill deeper into what these expenses are. In this post, we’ll break down our daily EC2 spend by the types of instances that are running.

Creating the RunBook

In our last post, we built a RunBook that used the AWS Cost and Usage report to break down unSkript’s costs product. By running the report daily, we could chart our daily cost spend per product, and if the change day over day exceeded a threshold, we could send an alert.

To study our daily EC2 usage and spend, we will use the same Cost and Usage report, and the RunBook is essentially the same – just with a different SQL query into the table. Rather than recreate the RunBook, we will simply Duplicate the RunBook in the unSkript UI, and save it with a new name:

To run a AWS Redshift query, we need a SecretArn, the SQL Query, AWS Region and Redshift Cluster and database details.

The RunBook generates these all for us, but we need to update the SqlQuery variable to query EC2 data:

sqlQuery = f"SELECT date_part(day, cast(lineitem_usagestartdate as date)) as day, product_instancetype,SUM(lineitem_usageamount)::numeric(37, 4) AS usage_hours, SUM((lineitem_unblendedcost)::numeric(37,4)) AS usage_cost FROM {tableName} WHERE length(lineitem_usagestartdate)&gt;8 AND product_productfamily = 'Compute Instance' AND pricing_unit IN ('Hours', 'Hrs') GROUP BY day, product_instancetype ORDER BY 1 DESC, 3 DESC, 2

The next few steps of this RunBook is unchanged from the Product Cost RunBook:

We take the Ann and the query – and run the query – then pull the results into the RunBook.

We must now change the chart – since the inputs are different. Our Columns are now “product_instancetype” and the y values are “usage_cost.”

Charting this data helps us see which instance types are costing the most money:

Our daily costs are very much proportional to size, with the 2xLarge and Large instances accounting for ~$20/day. It is interesting to note that earlier this month, our t2.large went down at the same time our t2.micro grew – this could possibly be interpreted as resizing of an EC2 instance that was too large.

Finally, we can build alerts to tell us if any of our costs jump by over $1 a day, or 10%. If the costs make a jump we can send the chart EC2 usage to Slack using the Send Image To Slack Action (coming this week to our Open Source). We also send this image every Monday for a historical record:

Conclusion

When it comes to FinOps – the team in charge of understanding and accounting for your cloud bill, one of the most critical tools to have is observability into your daily spend. If there is a large change – that might be ok – but it is good to know in advance, and double check that the increase is valid – thereby potentially avoiding a large bill at the end of the month.

In our previous post, we looked at all AWS products for daily cost spend, and in this post, we dug deeper to better understand our EC2 spend – the largest percentage of our AWS bill. Are you interested in trying out our alerting in unSkript? Try out our Open Source Docker build (and give us a star!), or sign up for a free cloud trial. Is there a segment of YOUR AWS bill that you’d like to investigate with us? Reach out in our Slack channel, and we’d be happy to help you create a RunBook for your use case!

Keeping your Cloud Costs in Check: Automated AWS Cost Charts and Alerting

Doug Sillars — Mon, 20 Mar 2023 16:03:51 +0000

Building and deploying infrastructure in the cloud is (by design) a very simple process. If a team is not being careful in their deployments, it can also become an expensive process. The interplay between finance teams and cloud teams has led to a new job function – FinOps. What is FinOps? This team (or team member) works with developer teams to understand cloud needs, negotiates better prices with cloud operators, and can help translate cloud expenses and needs to the finance team.

However, many companies working in the could don’t have the luxury of allocating a team member to understanding cloud costs, and it is left to the dev teams to do their best to mitigate surprise bill. Without a full-time FInOps professional, teams will need tooling to help them better understand and control their cloud bills.

The FinOps Institute has defined 6 domains for a FinOps tram:

Understanding Cloud Usage and Cost
Performance Tracking and Benchmarking
Real Time Decision Making
Cloud Rate Optimization
Cloud Usage Optimization
Organizational Alignment

In this post, we’ll describe a series of automated RunBooks that check the first three boxes (and help inform several others). What we’ll do is build automated reporting around the AWS Cost and Usage Report (CUR).

The AWS CUR

AWS’ Cost and Usage Report is a SQL report that breaks down your AWS spend in many different ways, and at a selected interval (hourly, daily, monthly). When generating a CUR report, a SQL file is placed into a S3 bucket at regular intervals. To set up a CUR report for your AWS Account, the AWS Documentation has a very nice tutorial. In this post, our CUR is updated daily, but for larger projects, you may want hourly granularity.

We set our CUR report to be sent into AWS Redshift. However, the daily updates are only added to the S3 file. To update our Redshift table, we must regularly take the file in SQL and update the database in Redshift.

We accomplish this by building a RunBook using a few new Actions:

When Making a RedShift Query, we need to know the Secret Manager ARN, the AWS Region, and the Redshift Cluster/database.

To get the Secret Arn, we use the AWS GET Secrets Manager ARNAction. This takes a secret name, and provides the ARN. (This does require Secrets Manager permission in your IAM credential.)

We then Create two SQL queries programatically. The table in Redshift is named awsbilling202303 (since it is currently March 2023). To ensure that the table name is always correct, we generate the query programmatically, so that the table name always has the format awsbilling<year><month>.

Next, we perform two SQL commands:

First, we TRUNCATE the table. This removes all of the rows, but keeps the columns:

truncate table awsbilling202303

Next, we COPY the rows from the SQL table in AWS:

copy awsbilling202303 from 's3://unskript-billing-doug/all/unskript-billing-doug/20230301-20230401/unskript-billing-doug-RedshiftManifest.json' credentials 'aws_iam_role=arn:aws:iam::<arn name>' region 'us-west-2' GZIP CSV IGNOREHEADER 1 TIMEFORMAT 'auto' manifest;

This query is provided in your S3 bucket.

With the RunBook created, we can schedule this RunBook to run daily, ensuring that the AWS table is always up to date.

NOTE: I am not a database expert. This is the “I have a hammer, so everything must be a nail” approach to updating the database. There is probably a more nuanced query that could be run.

Building Charts and Alerts

Now that the data is being populated into Redshift daily, we can begin exploring our data. In this second RunBook, we are going to extract the daily spent for each AWS service, plot the data, and create an alert for large changes in costs.

Our new RunBook begins the same way as our first RunBook – generating a SQL query and executing it at RedShift. This time we are querying Redshift for usage costs for every AWS product:

select lineitem_productcode, 
        date_part(day, cast(lineitem_usagestartdate as date)) as day, 
        SUM((lineitem_unblendedcost)::numeric(37,4)) as cost 
from awsbilling202303 
group by lineitem_productcode, day 
order by cost desc;

This query is then placed into a dataframe. Using this data, we can create a chart of our daily spend by AWS product:

Here is the same chart – just looking at the last seven days:

If we look back at the FinOps Institute bullet list, we are beginning the FinOps Institute bullets: Understanding Cloud Usage and Cost, as well as Tracking our Performance.

If we automate this RunBook to run daily (after the table is updated), we can generate this regular plot of our AWS Cloud Spend. In general, a daily chart with no change is not of great interest. But – if we can build an alert around our Cloud usage that finds significant increases in day to day cost – we can attach this chart to make the cost jump easy to identify.

Building an Alert

Every organization will have different thresholds for alerting. In the following code, we examine the 2 days previous (March 18 and 19) and look for increases of over 5%. Since many of the services in this chart are at very low spend rates, we add the additional filter that the change must be over $1. Looping over each service with the following:

if abs(todayCost-yesterdayCost) >1: 
   if delta >.05:
       #print( instance, delta,dfpivot.at[today, instance], dfpivot.at[yesterday, instance])
       bigchange[instance] = {"delta":delta, "todayCost":todayCost,"yesterdayCost":yesterdayCost}
       alertText = '@here There has been a large change in AWS Costs'
       alert = True
if date.today().weekday() == 0:
   alertText = 'Today is Monday, Here is the last week of AWS Costs'
   alert = True

If any of the changes in cost trigger this alert, we send an image on Slack with a ‘@here, there’s been a large change in AWS Spending” message. We also send the chart every day on Monday., so that there is a visual history of AWS spending.

When beginning to study and alert on your Cloud Spending it is important to start with simple reports and altering. This use case of daily spend by product, is a great start into our journey into the FinOps Institute’s next bullet points: Real Time Monitoring and Cloud Usage Optimization.

Conclusion

Many companies feel the monthly dread of “how big will my Cloud bill be this month?” By charting and alerting on your daily spend across all Cloud products, your team is less likely to be surprised by the beill at the end of the month. This data can also be used mitigate large changes that often result in bill “surprises.”

Using unSkript and the AWS Cost and Usage report, you can begin (or continue) your FinOps journey by better understanding where your costs are coming from and how they are changing day to day. Watch our blog for more posts on Cloud CostOps and how you can monitor your AWS bill.

AWS Service Quotas: Discovering Where you Stand

Doug Sillars — Fri, 17 Feb 2023 19:31:01 +0000

We’ve written a few posts in the last week about AWS Service Quotas. These are restrictions on services that are set by AWS (but can often be increased).

If our first post, we looked at New Actions in unSkript that can be used to determine quota values and request a quota increase. In this post, we’ll take the Actions a step further, and build an Action that compares the AWS quota to actual usage – generating an alert when a threshold is met.

Getting Started

To begin, we will need to consider how the Action will work. For any given service, we’ll need to query AWS at least twice:

Get the Quota Limit.
Determine the usage of a service.

Every query requires one call to complete Step 1. However, Step two can require many queries to complete the usage query. In the simplest case, we can do just one query:

Example: _Client VPN Endpoints per Region. _ If we query AWS for the list of endpoints in a region, we can simply get the length of the response to know how many endpoints exist.

However, there are times where there will be multiple queries:

Example: Routes per Client VPN Endpoint. In the first query, we get the list of VPN endpoints. In step 2, we must query every VPN endpoint to get the count of Routes. If there are 4 VPN endpoints, there will be a total of 5 calls made (On call to get the list of 4 VPN endpoints, and then one call to each of the four endpoints).

To account for these two options, we create an input Dictionary.

The Simple, one pass Dictionary

For the Describe AMIs call (only one Usage query is required), the Dict looks like this:

{'QuotaName':'AMIs','ServiceCode':'ec2','QuotaCode': 'L-B665C33B',
'ApiName': 'describe_images', 'ApiFilter' : '[]','ApiParam': 'Images',
'initialQuery': ''},

To get the Quota, we need the ServiceCode and the QuotaCode (If you need to obtain these variables, you can use the unSkript Action, or you can refer to the table in the unSkript Docs). The one usage API call will be made to the describe_images endpoint, and retrieve a list of Images. Counting this length gives us our usage.

The Two Pass Dictionary

To determine the Attachments per transit gateway, we must again get the quota from the Service Code and Quota Code. To get the count of attachments per transit gateway, we us the initalQuery array to make a first query.

The first query probes the describe_transit_gateways endpoint, to give a list of TransitGateways. _ In the second set of calls, we call the _describe_transit_gateway_attachments endpoint for each transit gateway. The filter has a string VARIABLE that is replaced with the TransitGatewayId for each gateway -ensuring that each call is made to a different transit gateway. We can then count the length of the response to find out how many attachments are in each transit gateway. If we have 12 transit gateways. we will have 12 usage reports.

{'QuotaName':'Attachments per transit gateway','ServiceCode':'ec2','QuotaCode': 'L-E0233F82',
'ApiName': 'describe_transit_gateway_attachments', 'ApiFilter' : '[{"Name": "transit-gateway-id","Values": ["VARIABLE"]}]',
'ApiParam': 'TransitGatewayAttachments',
'initialQuery': '["describe_transit_gateways","TransitGateways", "TransitGatewayId"]'},

Outliers

For most of our quota measurements, these two approaches work well. However, with over 2600 different quotas inside AWS, not all of them fit neatly into these two buckets. For example Multicast Network Interfaces per transit gateway requires 3 calls: Transit gateways -> Multicast Domains – > Domain attachments.

For others, there is custom code to iterate over. These require an extra if statement in the code to properly account for their usage.

Action Format

We can differentiate between the two types of query by looking at the ‘initialQuery’ parameter. If it is empty, we can do the Simple query, otherwise, do the double query (with a for loop that queries each initial result). For outliers, we can add specific code inside the if/else:

(this is simplified a bit from what actually runs):

for i in table: 
    #get quota 
    sq = sqClient.get_service_quota(ServiceCode=i.get('ServiceCode'),QuotaCode=i.get('QuotaCode')) 
    quotaValue =sq['Quota']['Value'] 

    #get usage     
    if i.get('initialQuery') = '': 
       res = aws_get_paginator(ec2Client, i.get('ApiName'), i.get('ApiParam'), Filters=filterList) 
       count = len(res) 
       percentage = count/quotaValue 
       combinedData = {'Quota Name': i.get('QuotaName'), 'Limit':quotaValue, 'used': count, 'percentage':percentage} 
       result.append( combinedData) 
       print(combinedData) else: res = aws_get_paginator(ec2Client, i.get('ApiName'), i.get('ApiParam'), Filters=filterList) 
       for j in res: #build the filter query with some simple substitutions res2 = aws_get_paginator(ec2Client, i.get('ApiName'), i.get('ApiParam'), Filters=filterList) count = len(res2) percentage = count/quotaValue objectResult = {j[initialQueryFilter] : count} quotaName = f"{i.get('QuotaName')} for {j[initialQueryFilter]}" combinedData = {'Quota Name': quotaName, 'Limit':quotaValue, 'used': count, 'percentage':percentage} result.append(combinedData)

Action Output

Once all of the values have been collected, the percentage utilized is compared to the warning percentage input. If the utilization is over the requested percentage, the Service data will be added to the output of the Action. With this information, the SRE responsible can decide the correct Action to take – either prune away some usage, or request an increase from AWS.

For example, testing all VPC Service quotas with a earning of 50% utilization gives the following data:

{'Instances': [{'Limit': 20.0,
                'Quota Name': 'VPCs Per Region',
                'percentage': 0.65,
                'used': 13},
               {'Limit': 20.0,
                'Quota Name': 'Internet gateways per Region',
                'percentage': 0.6,
                'used': 12},
               {'Limit': 5.0,
                'Quota Name': 'NAT gateways per Availability Zone',
                'percentage': 0.8,
                'used': 4},
               {'Limit': 50.0,
                'Quota Name': 'Routes per route table',
                'percentage': 0.5,
                'used': 25},
               {'Limit': 20.0,
                'Quota Name': 'Rules per network ACL',
                'percentage': 0.65,
                'used': 13}]}

Availability Today

As we publish this article, we have 2 Actions heading into unSkript:

A general AWS_ServiceQuota Compare Action that has the basic framework described above. This will likely require customization for each Quota you wish to test against.
AWS VPC Service Quota Warning. This Action takes all of the VPC service quotas (as of February 2023) and tests them against your infrastructure.

Coming Soon:

AWS EC2 Service Quota Warning. This Action will test your infrastructure against all EC2 Service Quotas, and warn you if you are approaching the quota threshold.

WE’re really excited to see how people use these Service Quota alerts in their infrastructure. If you have questions – feel free to reach out in our Slack Community. If you haven’t tried unSkript – try our OSS Docker Container, or use our free trial online!

AWS Service Quotas, or AWS has a LOT of Services!

Doug Sillars — Wed, 15 Feb 2023 18:03:21 +0000

In our recent post, we unveiled unSkript Actions that can query AWS Service Quotas. Service quotas are limits imposed by AWS on how many times a certain AWS feature can be used. Most of them are adjustable with a simple request, and our post showed how to determine your Service Quota values, AND request an increase using unSkript.

In this post, I thought it might be fun to dig into AWS Service Quotas a bit deeper, and get s general idea of how Service Quotas fit into the AWS landscape.

To get the Service Quota value, you need to know the Service Name, and the Quota Code. But how do you get these values?

AWS Service Names

We can get all of the AWS Service Names using the Service Node endpoint. Running this call, we find that there are 221 named services in AWS (as of Feb 15, 2023). AWS gets a lot of flak for their naming conventions, but with so many services, if course some are going to have sup-optimal names. Lucky for us, we’ll be using the ServiceCode, and not the Service Name, so “_ AWS Systems Manager Incident Manager Contacts ” is simply “ **_ssm-contacts** ” and “AWS IAM Identity Center (successor to AWS Single Sign-On)” is just “_ sso. _”

AWS Service Quotas

Next, we can run these 221 named services against the “List Service Quotas” endpoint to get all of the Service Quotas for all of the Services. Only 113 of AWS Services (51%) have features that have a service quota. Even with just half of services having quotas – there are a _ LOT _ of preset quotas in AWS: 2,629 of them in fact! (Feb 15,2023)

Service Code	Count of Quota Name
sagemaker	702
ec2	131
iotwireless	102
kinesisvideo	82
rekognition	80
personalize	66
cases	65
braket	62
elasticmapreduce	60
geo	60
comprehend	56
kms	53
lookoutmetrics	44
sns	44
logs	41
apigateway	38
iotcore	37
chime	36
forecast	33
ebs	32
glue	28
acm-pca	26
fsx	26
robomaker	24
rds	24
elasticloadbalancing	22
dataexchange	22
mediapackage	22
monitoring	22
cognito-idp	21
elasticfilesystem	21
vpc	21
fis	19
servicecatalog	17
omics	17
ssm-contacts	16
application-autoscaling	15
ssm	15
cassandra	14
frauddetector	14
imagebuilder	14
servicequotas	13
workspaces	13
textract	13
lex	13
events	12
amplify	11
appmesh	11
route53resolver	11
iottwinmaker	11
databrew	11
medialive	11
rolesanywhere	10
mgn	10
access-analyzer	9
eks	9
groundstation	9
nimble	9
dms	9
dynamodb	9
workspaces-web	8
ecr	8
ivschat	8
profile	8
batch	7
proton	7
mediastore	7
drs	6
appconfig	6
pinpoint	6
schemas	5
cloudformation	5
athena	5
m2	5
ivs	5
ec2-ipam	5
es	5
refactor-spaces	4
kafka	4
resiliencehub	4
app-integrations	4
network-firewall	4
resource-explorer-2	3
AWSCloudMap	3
qldb	3
ssm-sap	3
mediaconnect	3
auditmanager	3
sms	2
lambda	2
guardduty	2
ram	2
ses	2
autoscaling	2
fargate	2
cloudhsm	2
outposts	2
airflow	2
dlm	2
license-manager	2
ssm-guiconnect	1
connect-campaigns	1
connect	1
macie2	1
iotanalytics	1
emr-serverless	1
codeguru-profiler	1
autoscaling-plans	1
firehose	1
codeguru-reviewer	1
kinesis	1
grafana	1
ec2fastlaunch	1

As Machine Learning can be a very expensive process, it is no surprise that Amazon SageMaker leads the pack with over 700 service quotas. 2nd in line is an oldie but a goodie, Amazon’s EC2 (debuted in 2006!) with 131 quotas.

What do we know about quotas?

The longest quota name belongs to Rekognition, and it is quote a mouthful:

_ Transactions per second per account for the Amazon Rekognition Image personal protective equipment operation DetectProtectiveEquipment. _with a quota of 5. That’s a lot of words to say that the service can scan 5 frames per second to identify a helmet, face mask or gloves on anyone in the picture. This quota can be adjusted, if desired.

The largest Quota is ElasticFileSystem’s file size , weighing in at 52673613135872 bytes. (Which, if I did my math correctly, is 47.9 TB). This is a hard limit and cannot be adjusted.

The second largest quota is the Maximum number of rows in a dataset for Amazon Forecast, with a soft limit of 3 billion rows. You can request that this number be increased.

Quota adjustment

Of the 2,629 quotas, 2,003 can be adjusted (76%), and 626 (24%) cannot be changed.

Quota Units

Only 87 of our quotas have units (3.3%). 20 are time based, and the remaining 67 are data sizes (of varying magnitude):

Unit	Count
Millisecond	2
Second	18

These vary from 200 ms ( API Gateway Maximum Integration Timeout ) to 30 days: SageMaker’s Longest run time for an AutoML job from creation to termination. In case you were wondering, 30 days is also 2,592,000 seconds.)

When it comes to units, there’s nothing like arbitrarily multiplying by 1024 to change the units (and I see you MegaBits and your extra x8… but these are all throughput, so I’ll give that a pass).

Unit	Count
Bytes	11
Kilobytes	15
Megabits	4
Megabytes	13
Gigabytes	22
Terabytes	2

The smallest value is Lookout Metrics Value Length at 40 Bytes, and the largest is RDS Total storage for all DB instances at 100,000 GB (or 97 TB).

The winner for the oddest size measurement goes to Elasticfilesystem’s Throughput per NFC Client at 524.288 MegaBytes.

Summary

While looking at the giant list of AWS Service Quotas, I thought it might be fun to look at the data more closely. It remains to be seen whether the unSkript team will continue to let me use PivotTables to look at data.

More importantly, the list of Service quotas – with the Service Code and Quota Code are all in one table, and we have published the Feb 15, 2023 list in the unSkript Docs.

If you’re interested in learning more about unSkript, join our Slack Community, or check out our Open Source repo, you can run unSkript Open Source locally with Docker!

AWS Service Quotas: What are they, and how can I increase them?

Doug Sillars — Tue, 14 Feb 2023 00:35:30 +0000

Everywhere we go, there are limits – speed limits, weight limits, capacity limits. AWS is no different. For many of the services offered by AWS, there is a service quota limit – the maximum number of that feature each account holder is allowed to use.

Can we determine the limits imposed by AWS?

Yes, the AWS service-quotas API endpoint can tell us what quotas exist and their values?

Can we ask for a limit increase?

For most of the quotas, you can ask for your limit to be increased.

In this post, we’ll use Actions in unSkript to automate work around Service Quotas – determining limits, and asking for an increase to a quota. All of these Actions are soon to be a part of the unSkript Aweseome Runbooks GitHub repository. They’ll also be included in all instances on unSkript.

Probing the AWS Service Quota Surface

The first step in understanding AWS Service Quotas is to look at the API. There are a few interesting endpoints, but in order to query a specific service or request a quota increase, we will need three items:

Region
Service Code
Quota Code

We probably have a good idea what AWS Region our stack is deployed to – that’s the easy part. We still need to understand and figure out the Service Code and the Quota Code for our service.

Service Codes

Service codes describe the top level services that AWS offers (think S3, EC2, etc.) To obtain the AWS Service Codes, we can utilize the list-services endpoint in our Action named AWS Get All Service Names v1 :

def aws_get_all_service_names(handle, region:str) -> List:
    sqClient = handle.client('service-quotas',region_name=region)
    resPaginate = aws_get_paginator(sqClient,'list_services','Services',PaginationConfig={
        'MaxItems': 1000,
        'PageSize': 100
        })

    #res = sqClient.list_services(MaxResults = 100)
    return resPaginate

This lists all of the AWS Services by name and their service code. At the time of this writing, there are 220 ( edit – it is now 221!) services in the output.

Quota Codes

Quota codes are available in the AWS Console interface, but it requires a lot of digging to get them (and sometimes it is easiest to find them in the url string). An easier way is via the API:

The list_service_quotas endpoint takes a Service Code (from the first list), and outputs all of the service quotas for that service. The AWS Get Service Quotas for a Service v1 Action obtains the codes, given a Service Code

Setting the Service Code to “ec2”, and the Region to “us-west-2”, we get 129 different Quotas (and the quota code for each one of them).

    sqClient = handle.client('service-quotas',region_name=region)
    resPaginate = aws_get_paginator(sqClient,'list_service_quotas','Quotas',
        ServiceCode=service_code,
        PaginationConfig={
            'MaxItems': 1000,
            'PageSize': 100
        })

    #res = sqClient.list_services(MaxResults = 100)
    return resPaginate

A sample output

Here is a sample of the EC2 output:

{'ServiceCode': 'ec2', 'ServiceName': 'Amazon Elastic Compute Cloud (Amazon EC2)', 'QuotaArn': 'arn:aws:servicequotas:us-west-2:100498623390:ec2/L-70015FFA', 'QuotaCode': 'L-70015FFA', 'QuotaName': 'AMI sharing', 'Value': 1000.0, 'Unit': 'None', 'Adjustable': True, 'GlobalQuota': False}

This says that the limit for the number of Amazon Machine Images (AMIs) that you can share is 1000.

Requesting a Quota Increase

What if we wanted to share 1001 AMIs? We can request a quota increase via API. In the several attempts I have made, they were all grated automatically – but not immediately. Using the request service quota increase endpoint in the AWS Request Service Increase Action adds your request to the AWS queue for processing:

def aws_get_service_quotas(handle, service_code:str, quota_code:str, new_quota:float,region:str) -> Dict:
    sqClient = handle.client('service-quotas',region_name=region)
    res = sqClient.request_service_quota_increase(
        ServiceCode=service_code,
        QuotaCode=quota_code,

        DesiredValue=new_quota)

    #res = sqClient.list_services(MaxResults = 100)
    return res

This Action has 3 inputs – the Service_code, the quota_code, and the integer value that you would like the quota changed to.

Conclusion

In this post, we built Actions using unSkript to learn what service quotas exist in your AWS Account, and how to update them. The Actions described in this post are built into unSkript’s automation engine, allowing you to build custom RunBooks around service quotas in your AWS environment.

Are you interested in learning more? Try out our Open Source Docker build. Instructions can be found in the GitHub Readme file. If you have questions, join our Slack channel, where the community will be happy to help you!

Copying AWS Amazon Machine Images Across Regions with unSkript

Doug Sillars — Tue, 24 Jan 2023 20:29:29 +0000

When you are building a distributed platform, You’ll need to regularly update the machines that you have deployed around the world. With Amazon Web Services (AWS), one way to do this is to deploy the updated machine in one region, and create an Amazon Machine Image (AMI) of that server. Then, by copying that AMI to different AWS regions, you can easily deploy an identical server anywhere around the world.

Copying AMIs across regions is possible to do via the AWS console UI, or using the AWS command line. At unSkript, we are working to remove the manual toil of running scripts against the command line, or performing multi-step manual processes in the UI. With hundreds of pre-built connectors and Actions to perform common tasks with the most popular cloud services – it is easy to get started with unSkript in just minutes. Try it today – either with ourOpen Source Docker image that you can run on-premises, or with afree trial of our Cloud offering.

Copying AMIs across regions

In your install of unSkript, there will be a RunBook called “Copy AMI to All Given AWS Regions.” This RunBook is pre-configured to perform the AMI copy with just the click of a button. When you open this RunBook (or import your copy into your work area), you’ll only have a few steps before you can begin copying your AMIs.

At the top of the page, there is a Parameters drop down, where the input parameters for the RunBook are entered:

In the screenshot above, we have an ami_id (with pixelated name) with a source_region of us-west-2. We are looking to copy this AMI to the destination_regions us-east1 and us-east-2. These can be edited with the Edit Value button. When you run this RunBook via the UI – you will be give the opportunity to insert different values.

Configuring the Actions

Each step of the RunBook is completed by an Action. We will need to quickly configure each Action with credentials that allow unSkript to connect to AWS. If you do not yet have AWS Credentials in unSkript, follow the link to set up your credentials. To Configure your action, click the Configurations button. On the right side, select the AWS credential you’d like the RunBook to connect with:

Connecting with the DevRole credential

Repeat this for all AWS Actions in the RunBook:

Get all AWS Regions
Get AMI Name
Copy AMI to Other Regions

Once the credentials are set up, save your RunBook. It can now be run from this page in two ways: Run XRunBook button: or interactively – running one Action at a time. But it can now also be run via the RunBook listing page, or using our API.

Here is the interactive output of the final Action, where the AMI image was copied into us-east-1 and us-east-2.

Summary

In this post, we utilized unSkript to automate the copying of AMI’s across AWS regions. Rather than build a RunBook ourselves, we were able to utilize a pre-built RunBook, and simply configure the RuBook for our use case. With just a few steps, we were able to authenticate this RunBook to run in our AWS instance, and copy AMI images from one region to many other regions. By automating such repetitive tasks, unSkript alleviates DevOps toil of repetitive tasks, allowing the team to focus on more important projects.

Give us a try today, and if you like what you see, give us a star on GitHub! Questions? Join our Slack Community – we’d be happy to help.

Managing your Cloud Costs with CloudOps Automation Part 1: Identifying Your Resources with Tags

Doug Sillars — Thu, 15 Dec 2022 19:01:48 +0000

Moving systems to the cloud makes a lot of sense operationally – letting the experts take care of the infrastructure, and let us build what we need to make our company successful.

But this comes at a substantial downside – your monthly cloud bill. Cloud providers have made it insanely easy to spin up new servers and features, but without careful auditing – it is *very* easy to leave money on the table due to unused or improperly sized resources remaining active on our cloud provider. In this series of posts, we’ll use unSkript to uncover unused assets in your cloud account, and then either alert the team of their existence, or remove them automatically.

This of course leads to a catch 22 – how do you know what is safe to remove – and what will bring down production? In order to better identify our cloud resources, we need a good tagging strategy.

So, to kick off our cost savings series of posts, we’ll begin with a discussion on tagging your cloud resources.

Why Tag your resources?

Tags are key:value pairs that describe your cloud function. With AWS, you can use any value for your key – giving the ultimate in customization. AWS has a number of best practices for tagging your resources.

Tagging allows us to easily identify our cloud components and quickly determine what the components do. AWS recommends “overtagging” vs “undertagging.” In many ways, this is the CloudOps analogy to commenting code. With a ‘well-tagged” set of resources, auditing each instance becomes easier.

Perhaps most importantly – it becomes easy to find resources that are no longer needed, allowing for them to be turned off keeping your cloud bill in check.

Tag Strategy

There is no definitive set of tags that should be used, but here are some that are often discussed:

Environments: If your different environments are all in the same Cloud, labeling each object with an Environment key, and values like development, staging, production help you understand where in the deployment process your instances lie.
*Department: * Tagging the department that owns/controls the resource has a number of important features:
2. The team can track their cloud installs.
3. If there is a problem with the instance, the correct team can be easily identified and notified that there is an issue.
*Cost center: * Identify teams with higher spends. More easily break down budgeting for cloud billing.
*Expiration: * When building out a new system, you may deploy a number of instances for testing. By setting a sunset date, you can remove any worry of accidentally leaving a cloud instance live – they will all shut down within a few days or weeks.

Building your tagging strategy.

If you have not yet built out your strategy, you probably have dozens (hundreds) of untagged cloud objects. Since there is no way for one person or team to know what each instance is, or even if it is still in use, we need to add tags.

So the first step is to identify each object, and contact the owners to add tags to the instances.

Of course, it would be best to give your team an automated way to add tags to their instances, so that they do not lose a lot of bandwidth complying with the new requirements (with the added benefit that an easy onboarding will make your new tagging policy go faster with less friction. Here’s how we have done this at unSkript (using unSkript xRunBooks and Actions, or course).

Step 1 Find resources with zero tags

unSkript has a built in action called “ AWS Get Untagged Resources.” This Action calls all EC2 instances that have no tags attached to them, We can search for this Action and drag it into our workflow. (In our Free Sandbox, create a new xRunBook, and then search for the action)

Connect your AWS Credentials (learn how to create a connection to AWS), and add your Region (you can either change the value in the configurations to the right, OR change the parameters in the top menu – this refers to the variable Region). When run, this Action gives a list if instanceIds that have no tags. We’d like a bit more information, so we’ll edit the xRunBook to look like this:

def aws\_get\_untagged\_resources(handle, region: str) -> List:



    print("region",region)

    ec2Client = handle.client('ec2', region\_name=region)

    #res = aws\_get\_paginator(ec2Client, "describe\_instances", "Reservations")

    res = aws\_get\_paginator(ec2Client, "describe\_instances", "Reservations")

    result = []

    for reservation in res:

        for instance in reservation['Instances']:       

            try:

                #has tags

                tagged\_instance = instance['Tags']

            except Exception as e:

                #no tags

                result.append({"instance":instance['InstanceId'],"type":instance['InstanceType'],"imageId":instance['ImageId'], "launched":instance['LaunchTime'] })

    return result

We make these changes to give a bit more information about each instance that is untagged. For example:

[{‘imageId’: ‘ami-094125af156557ca2’,

‘instance’: ‘i-049b54f373769f51b’,

‘launched’: datetime.datetime(2022, 12, 14, 17, 48, 49, tzinfo=tzlocal()),

‘type’: ‘m1.small’},

Now we can reach out to the rest of the team to see if anyone knows about this m1.small instance launched on 12/14/22 from a specific AMI.

Step 2: Add tags to found instances:

We now have a list of all of the instanceIds that have no tags. Now we can use a new action that attaches tags to an EC2 instance to begin the process of bringing the instance into tagging compliance.

This action has 3 inputs: _instanceId_, _Tag\_Key_, and _Tag\_Value_.def aws\_tag\_resources(handle, instanceId: str, tag\_key: str, tag\_value: str, region: str) -> Dict:

    ec2Client = handle.client('ec2', region\_name=region)
    result = {}
    try:
        response = ec2Client.create\_tags(
            Resources=[
                instanceId
            ],
            Tags=[
                {
                    'Key': tag\_key,
                    'Value': tag\_value
                },
            ]
        )
        result = response
    except Exception as error:
        result["error"] = error
    return result

Running this Action adds the key:value tag into the EC2 instance.

Step 3: Compliance check

Finally, we’ll build one last Action that checks all tag Keys against the required list of keys, and returns those instances that are mossing a required tag:

def aws\_get\_resources\_out\_of\_compliance(handle, region: str, requiredTags: list) -> List:

    ec2Client = handle.client('ec2', region\_name=region)
    #res = aws\_get\_paginator(ec2Client, "describe\_instances", "Reservations")
    res = aws\_get\_paginator(ec2Client, "describe\_instances", "Reservations")
    result = []
    for reservation in res:
        for instance in reservation['Instances']:       
            try:
                #has tags
                allTags = True
                keyList = []
                tagged\_instance = instance['Tags']
                #print(tagged\_instance)
                #get all the keys for the instance
                for kv in tagged\_instance:
                    key = kv["Key"]
                    keyList.append(key)
                #see if the required tags are represented in the keylist
                #if they are not - the instance is not in compliance
                for required in requiredTags:
                        if required not in keyList:
                            allTags = False
                if not allTags:
                    # instance is not in compliance
                    result.append({"instance":instance['InstanceId'],"type":instance['InstanceType'],"imageId":instance['ImageId'], "launched":instance['LaunchTime'], "tags": tagged\_instance})

            except Exception as e:
                #no tags               result.append({"instance":instance['InstanceId'],"type":instance['InstanceType'],"imageId":instance['ImageId'], "launched":instance['LaunchTime'], "tags": []})
    return result

This Action reads in a list of required keys, and if an instance does not have all of them – it is returned in an out of compliance list.

Conclusion

It has been shown that tagging cloud instances makes troubleshooting faster. It also helps you identify cloud objects that are no longer in use – helping you to reduce your cloud bill. For these reasons it makes sense to create a tagging requirement for all instances.

In this post, we have created a series of Actions that will help you simplify the transition process of bringing all of your existing cloud objects into tagging compliance.

Feel free to try these Actions in our Free Sandbox, or using our Docker install. The actions used in this post will soon be available on Github in the xRunBook Add Mandatory Tags to EC2. Please reach out if you’d like a copy earlier!

Cloud Ops Auto Remediation: A Holiday Allegory

Doug Sillars — Wed, 07 Dec 2022 16:04:45 +0000

Auto remediations are tools that respond to events with automations able to fix, or remediate, the underlying condition. Building a demo that features auto remediation fix is hard, because generally modern infrastructure is resilient so keeping it in an error state is difficult. In order to feature an auto-remediation example, we’re going to get a little creative.

A Holiday Auto Remediation

I am writing this post in the first week of December, and let’s face it — holiday music is pretty inescapable this time of year. To describe the auto remediation that we will fix, let’s turn to one of the greatest fiction writers of our time:

As Mr. King has clearly presented — everyone has a least favorite holiday song, and the playing of this song can . During the holiday season, it is pretty common to use playlists generated by other users on Spotify. Wouldn’t it be great if we could auto remediate the playing of our disliked holiday songs automatically — without any intervention?

This is a great example of building auto remediation — an xRunBook in unSkript that identifies the song playing, and if there is an “error state” (a song on the blocklist), the RunBook will correct this issue without human intervention.

Auto Remediation in Spotify

In this post, we will build an auto remediation RunBook for your Spotify account.

We will create a Spotify app that connects with unSkript.
Curate a block list of songs.
We will build a xRunBook that checks to see if Spotify is playing.
If Spotify is playing a song on the block list, unSkript will automatically remediate the issue by skipping to the next track.
We can then place this xRunBook on a schedule to run every minute- ensuring that we’ll only hear a few seconds of our least favourite songs (at least when listening on *our* Spotify account). Note: This last step is only possible in the Open Source Docker build, or in a SAAS install of unSkript. This won’t work in the sandbox

Let’s get started:

Build a Spotify app

In order to interface with Spotify, we create an app at Spotify.

When you create an app, you’ll need to submit a redirect url. I used unskript.com in my example. Once you generate the app, you’ll get a clientId and clientSecret that you will need later to interface with your xRunBook. You can also add your account email as an authorized user of the application:

The name and description of my Spotify application gives away *my* reasoning for this app.

Building the xRunBook

NOTE: For full automation, install unSkript locally using Docker. If you just want to follow along the tutorial, you can use our free Sandbox to run the xRunBook manually. (If you are using the Sandbox, complete the tutorial.)

Now we are ready to create our xRunBook.

In Sandbox, Click xRunBook, and then “+Create”, and connect your RunBook to the proxy you created in the tutorial.
In Docker, there are instructions to create a new xRunBook in the README file.

xRunBooks are based on Jupyter Notebooks. Each action is an independent Python application. When the RunBook is executed, each Action is run in order. I have pasted the code for each action below. You’ll need to create a new Action for each code snippet. Do this by Clicking the + Add button at the top, and choosing “Action.”

Action 1: Install Spotipy

We’re going to use a Python library to interact with Spotify. In your first action:

!pip install spotipy - quiet

Action 2: Add our ClientIds.

In the OSS, and Sandbox, there is no built in Secret vault. so we’ll put them here. Make sure the redirect url matches what you placed in your Spotify application/

client_id = "<client ID from Spotify>"
client_secret="<secret from Spotify"
client_redirect="https://unskript.com"

Action 3: Connect to Redis

Note: not needed for Docker, and will not work for Sandbox, but this will work in unSkript SAAS:

import redis
redis = redis.Redis(host='<redis-host>', port=6379, db=0)
redis.set('foo', 'bar')
True
redis.get('foo')
b'bar'
print(redis)

Spotipy stores the authentication in a local cache — which is fine in the unSkript Docker instance.

The Sandbox does not have a local cache, nor a Redis instance, so this xRunBook cannot be run on a schedule as a result.

In our SAAS version, a Redis database can be attached to the xRunBook to ensure that the credentials are stored locally, and can be reused.

Action 4: Find SongIDs

Let’s do some searches in the Spotify database to extract the songIds we want to add to our blocklist. Here we are searching for songs sung by Mariah Carey:

import spotipy
from spotipy.oauth2 import SpotifyClientCredentials

scope = "user-library-read user-modify-playback-state user-read-playback-state"

sp = spotipy.Spotify(auth_manager=SpotifyClientCredentials(client_id=client_id,
                                               client_secret=client_secret))

results = sp.search(q='mariah carey', limit=5)
#print(results)
for idx, track in enumerate(results['tracks']['items']):
    print(idx, track['artists'][0]['name'], " ", track['name'], " " ,track['id'])

The results for this Action are:

0 Mariah Carey All I Want for Christmas Is You 0bYg9bo50gSsH3LtXe2SQn
1 Mariah Carey Fantasy 6xkryXuiZU360Lngd4sx13
2 Mariah Carey Christmas (Baby Please Come Home) 3PIDciSFdrQxSQSihim3hN
3 Mariah Carey We Belong Together 3LmvfNUQtglbTrydsdIqFU
4 Mariah Carey Fantasy (feat. O.D.B.) 2itAOPLerxnnc8KXHMqPWu

We want to (ok, I want to) block the song with ID: 0bYg9bo50gSsH3LtXe2SQn.

Action 5: Build a block list

This Action defines the array of songs we want to block.

songList = ["0bYg9bo50gSsH3LtXe2SQn", 
            "4iHNK0tOyZPYnBU7nGAgpQ",
            "0SorhWEyl6wkQ6vYAQt2D0"]

Action 6: Authenticate the user

This requests access for a user at Spotify. We need to read the playback state (is Spotify playing?), and we need permission to modify the state (skip the song!).

We have commented out the Redis cache_handler. If you are using the SAAS version of unSkript — remove the comment.

import spotipy
from spotipy.oauth2 import SpotifyOAuth
import json

scope = "user-library-read user-modify-playback-state user-read-playback-state user-read-recently-played"

spUser = spotipy.Spotify(auth_manager=SpotifyOAuth(client_id=client_id,
                                               client_secret=client_secret,
                                               redirect_uri=client_redirect,
                                              # cache_handler=spotipy.cache_handler.RedisCacheHandler(redis),
                                               scope=scope,
                                               open_browser=False))

Action 7: Skip the track

This step will be interactive. When this action is run, you’ll be asked to visit a url (that does a redirect) and then paste in the redirected url. This url has your authentication token in it. Once this is done, a token is stored locally that is valid for one hour. Every time the xRunBook is run, the token is re-authenticated for another hour.

#get current track
currentTrack = spUser.current_user_playing_track()

## only test the playback if there is currently a song playing
if currentTrack is not None:
    #
    track = currentTrack["item"]["uri"]
    print(track)
    #remove 'spotify:track:' from front of string to get the ID
    track = track[14:]
    print(track)

    # all i want for christmas is you. spotify:track:0bYg9bo50gSsH3LtXe2SQn
    songs_i_hate = songList

    for song in songs_i_hate:
        print("song", song)
        if track == song:
          print("ahhh save us")
          spUser.next_track()
          break
        else:
          print("its all good")
else:
    print("the music is off")

This completes the xRunBook creation. Save the RunBook (by closing it, and then selecting Save)

Scheduling the RunBook

In the SAAS (and Sandbox), it is possible to schedule your xRunbooks. By scheduling this xRunBook to run every minute — you can ensuyre that you’ll never have to listen to more than 59 seconds of the songs you dislike.

A Holiday Allegory

An Allegory is a metaphor that symbolizes an idea or message. Many stories told at Christmas time are allegories (like Dicken’s a Christmas Carol). In this post, we have used Spotify playlists as an allegory to a Cloud System in distress. By auto-skipping songs, our unSkript RunBook is automatically solving a situation without involving a human (potentially paging them out of a “long winder’s nap.”)

By building a “Skip the horrible track” auto remediation, we show the power of unSkript, and also potentially save the family Christmas by avoiding arguments over the Christmas playlist.

Interested in learning more about how unSkript can help you build internal auto remediation tooling for your team? Check out our free trial, star our GitHub repository of xRunBooks and Actions, or join our Slack Community!

SHH! Conductor has secrets!

Doug Sillars — Tue, 09 Aug 2022 00:00:00 +0000

We are really excited to announce the latest feature to Orkes' cloud hosted version of Netflix Conductor. It is now no longer a secret - we support the use of secrets in your workflow definitions! Now you can be certain that your secret keys, tokens and values that you use in your workflows are secure!

What do you mean by secrets?

In many of applications today, interaction with third party applications is common. Typically this will require some form of authentication to gain access. When you are coding, there is a concept of a local secure store where sensitive values are kept (and thus not shared to GitHub etc.) This prevents accidental disclosure of your secrets when posting code to GitHub or when sharing your code to other teams.

Until now, there has been no way to securely use any sensitive value in a Conductor workflow. Just about every developer has a story of accidentally posting a sensitive value on GitHub. Here's my story of accidentally sharing a sensitive value with a Conductor workflow:

In the order fulfillment codelab, the failure workflow has a Slack token that is unique, and if publicly accessible could be used to SPAM a Slack channel. When writing the tutorial, I shared the task definition in the docs - with the Slack token.

Slack caught this:

Super embarrassing, but no serious consequences (in this instance).

Don't let this happen to you!

In Orkes hosted instances of Netflix Conductor, we now feature secrets. You can save your secret on the Conductor server, and Conductor will use the value when required, but will not expose the value in any outputs from the workflow.

It is a very easy setup - simply login to your instance of Netflix Conductor at Orkes (or try our Playground for free!). In the left navigation, click Secrets. This will lead to a table of your secrets (which is probably empty).

Click Add Secret, give it a name, paste in your value, and press save. That's all there is to it.

Using your secret

In Conductor workflows, secrets use a similar format to other variables. For example, to reference an input variable called address you'd use ${workflow.input.address}.

If you had a secret called Stripe_api_key, you reference this value with the variable ${workflow.secrets.Stripe_api_key}.

An example

Accessing GitHubs APIs requires an API token. In the following HTTP task, I call a GitHub API, and can reference the secret Doug_github for the authorization header.

{ "name": "Get_repo_details", "taskReferenceName": "Get_repo_details_ref", "inputParameters": { "http_request": { "uri": "https://api.github.com/repos/${workflow.input.gh_account}/${workflow.input.gh_repo}", "method": "GET", "headers": { "Authorization": "token ${workflow.secrets.Doug_github}", "Accept": "application/vnd.github.v3.star+json" }, "readTimeOut": 2000, "connectionTimeOut": 2000 } }, "type": "HTTP", "decisionCases": {}, "defaultCase": [], "forkTasks": [], "startDelay": 0, "joinOn": [], "optional": false, "defaultExclusiveJoinTask": [], "asyncComplete": false, "loopOver": [] }

When this workflow is run, other variables are replaced, but the value of the secret remains a secret. Note that in the uri, ${workflow.input.gh_account}/${workflow.input.gh_repo} is replaced with netflix/conductor, but the authorization header remains obfuscated.

{ "headers": { "Authorization": "token ${workflow.secrets.Doug_github}", "Accept": "application/vnd.github.v3.star+json" }, "method": "GET", "readTimeOut": 2000, "uri": "https://api.github.com/repos/netflix/conductor", "connectionTimeOut": 2000}

Conclusion

Secrets have been one of the most requested features for Netflix Conductor when we speak to developers, and for that reason we're excited to announce this launch. We cannot wait to hear about how this release is making workflow development more secure and opening new avenues of development - now that these values can be securely stored.

Give them a try in the Orkes Playground, and we w ould love to hear what you think in our Slack or Discord communities

DEV Community: Doug Sillars

Database branching in Django apps using GitHub actions

Setup

Code repo

Digital Ocean

GitHub Secrets

Digital Ocean app spec files

Create NeonDB branch

Delete NeonDB branch

Conclusion

Automating Developer Relations Metrics with Low Code RunBooks

Collecting the Data

Built in Actions

Database Queries

REST API

Storing & reporting our data

Progressive enhancement

Scheduling

Summary

Automating the GitHub *Nudge*

Stale Issues

Issues without an assignee

Issues with an assignee

Stale Pull Requests

The actual “nudge”

Automating the Nudge!

Cloud Costs: Charting Daily EC2 Usage and Cost

Creating the RunBook

Conclusion

Keeping your Cloud Costs in Check: Automated AWS Cost Charts and Alerting

The AWS CUR

Building Charts and Alerts

Building an Alert

Conclusion

AWS Service Quotas: Discovering Where you Stand

Getting Started

The Simple, one pass Dictionary

The Two Pass Dictionary

Outliers

Action Format

Action Output

Availability Today

AWS Service Quotas, or AWS has a LOT of Services!

AWS Service Names

AWS Service Quotas

What do we know about quotas?

Quota adjustment

Quota Units

Summary

AWS Service Quotas: What are they, and how can I increase them?

*Can we determine the limits imposed by AWS? *

*Can we ask for a limit increase? *

Probing the AWS Service Quota Surface

Service Codes

Quota Codes

A sample output

Requesting a Quota Increase

Conclusion

Copying AWS Amazon Machine Images Across Regions with unSkript

Copying AMIs across regions

Configuring the Actions

Summary

Managing your Cloud Costs with CloudOps Automation Part 1: Identifying Your Resources with Tags

Why Tag your resources?

Tag Strategy

Building your tagging strategy.

Step 1 Find resources with zero tags

Step 2: Add tags to found instances:

Step 3: Compliance check

Conclusion

Cloud Ops Auto Remediation: A Holiday Allegory

A Holiday Auto Remediation

Auto Remediation in Spotify

Build a Spotify app

Building the xRunBook

A Holiday Allegory

SHH! Conductor has secrets!

What do you mean by secrets?​

Don't let this happen to you! ​

Using your secret​

Automating the GitHub Nudge

Can we determine the limits imposed by AWS?

Can we ask for a limit increase?

What do you mean by secrets?

Don't let this happen to you!

Using your secret

An example

Conclusion