DEV Community: Sathyajith Bhat

Bulk tagging all instances in an Auto Scaling Group (ASG) using AWS CLI and JMESPath Expressions

Sathyajith Bhat — Mon, 04 Sep 2023 00:00:00 +0000

Cover image generated by Stable Diffusion Online with the prompt: applying name tags to a server in a data center.

Why?

I wanted to tag all the instances in an AWS Auto Scaling Group (ASG) with some tags that would be used for reporting. The first order of business was to update the ASG with the relevant tags. This was done by applying the tag to the ASG using Infrastructure as Code (Terraform, in this case) so that all new instances launched would get the tag. However, the existing nodes would not inherit the tags.

While I could recycle the nodes, instance refresh makes it that much easier. Still, I figured bulk tagging the instances in the ASG was a simpler way to set the tags.

Filtering with AWS CLI

Since I wanted to tag instances of a specific ASG, I tried to use jq on the output of the aws ec2 describe-auto-scaling-groups command and further slice and dice the data. This was getting too painful, so I started to dig into AWS CLI’s docs and found out that the AWS CLI can apply client-side filtering using the --query option and server-side filtering using the --filter option.

Querying Auto Scaling Group Instances

The aws autoscaling describe-auto-scaling-groups command has a response structure as shown below:

[
    {
        "AutoScalingGroupName": "name",
        "AutoScalingGroupARN": "arn",
        "LaunchConfigurationName": "lc-name",
        "MinSize": 1,
        "MaxSize": 1,
        "DesiredCapacity": 1,
        "DefaultCooldown": 300,
        "AvailabilityZones": [
            "ap-southeast-1a",
            "ap-southeast-1c"
        ],
        "LoadBalancerNames": [],
        "TargetGroupARNs": [],
        "HealthCheckType": "EC2",
        "HealthCheckGracePeriod": 0,
        "Instances": [
            {
                "InstanceId": "",
                "InstanceType": "",
                "AvailabilityZone": "",
                "LifecycleState": "",
                "HealthStatus": "",
                "LaunchConfigurationName": "",
                "ProtectedFromScaleIn": false
            }
        ],
        "CreatedTime": "",
        "SuspendedProcesses": [],
        "VPCZoneIdentifier": "",
        "EnabledMetrics": [],
        "Tags": [],
        "TerminationPolicies": [
            "Default"
        ],
        "NewInstancesProtectedFromScaleIn": false,
        "ServiceLinkedRoleARN": "arn",
        "TrafficSources": []
    }
]

To query, the AWS CLI uses expressions created using JMESPath Syntax. From the response above, we want to select the instance IDs of a specific Auto Scaling Group. In JMESPath query, the question mark ? is used to filter and select elements. Thus, to filter based on AutoScaling Group name, the JMESPath Expression would be as shown below, replacing asg-name with the actual name of the Auto Scaling Group:

AutoScalingGroups[?AutoScalingGroupName==`asg-name`]

Filtering Instances Based on Auto Scaling Group

Since I further want to select only the instance IDs, the JMESPath expression of the instance ID .Instances[*].InstanceId can be chained to the above query condition. Thus, the command to fetch the instance IDs from a specific Auto Scaling Group becomes as shown below, taking care to replace region and asg-name with the region name and the ASG name, respectively.

aws autoscaling describe-auto-scaling-groups --region region --query 'AutoScalingGroups[?AutoScalingGroupName==`asg-name`].Instances[*].InstanceId'

Now that there’s a list of instance IDs, these IDs can be passed over to the aws ec2 create-tags command to apply the tag to all the instances.

aws ec2 create-tags --region ap-southeast-1 --tags Key=tagKey,Value=tagValue --resources $(aws autoscaling describe-auto-scaling-groups --region region --query 'AutoScalingGroups[?AutoScalingGroupName==`asg-name`].Instances[*].InstanceId' --output text)

Verifying the results

Applying what I learned above, I verified that all the instances have been correctly tagged by using the below command, taking care to replace region and asg-name with the region name and the ASG name, respectively.

aws ec2 describe-instances --region ap-southeast-1 --filters "Name=tag:aws:autoscaling:groupName,Values='asg-name'" --query "Reservations[].Instances[].{Instance:InstanceId,Name:Tags[?Key=='Name']|[0].Value}" --output table

----------------------
| DescribeInstances  |
+--------------------+--------------+
| Instance           | Name         |
+-------------------+---------------+
| i-0xdeadbeef456242 | asg-nme-1234 |
| i-0xdeadbeef454142 | asg-nme-1235 |
+-----------------------------------+

In the above command, the --filter option applies server-side filtering and fetches the instances that have the tag, tag:aws:autoscaling:groupName as the tag key and asg-name as the tag value. This is then chained with the client-side filter --query to display the instance ID and the name of the instance by fetching the tag with key Name.

Self-hosting FreshRSS (for free) on Fly.io in under 10 minutes

Sathyajith Bhat — Wed, 04 Jan 2023 00:00:00 +0000

FreshRSS is a free, self-hostable RSS feeds aggregator. Fly.io is a super amazing platform that runs application servers close to end users.

Fly.io can take Docker images (or Dockerfiles, or buildpacks) and boot them into Firecracker powered microVMs. When you deploy with Fly.io, you get an Anycast IP, along with TLS offloading by them. The Firecracker microVMs can scale up & down per traffic, and it comes with a pretty generous free tier that includes

Up to 3 shared-cpu-1x 256MB VMs
3GB of total persistent volume storage
160GB outbound data transfer

Since FreshRSS comes with a Docker image, adapting it to fly.io was as straightforward as pointing to FreshRSS' Docker image. These are the steps if you wish to run FreshRSS on your fly.io account

Sign up for Fly.io or sign-in
Install flyctl
Sign into your Fly.io account by typing flyctl auth login
Create a fly.toml file similar to the one in my repo. Make sure to update the appname to a unique name.
Review the environment variables listed in the [env] section and update if required. You can check FreshRSS' docs on the environment variables that can be updated
Create a volume for persisting data using the command fly volumes create freshrss_data --size 1
- In this command, we create a volume of size 1GB. This can be increased later, so I selected the lowest possible number, as flyctl expects size in GB and doesn't accept fractional numbers.
Save the fly.toml file. From the same directory where fly.toml is present, deploy FreshRSS the application using fly launch. Fly.io will pull the Docker image and launch the VM. The output should be as shown below:


fly launch
An existing fly.toml file was found for app freshrss
App is not running, deploy...
Deploying freshrss
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 80
Searching for image 'freshrss/freshrss' remotely...
image found: img_lj9x4d7jkwe4wo1k
Image: registry-1.docker.io/freshrss/freshrss:latest
Image size: 75 MB
==> Creating release
Release v1 created

You can detach the terminal anytime without stopping the deployment
Monitoring Deployment

1 desired, 1 placed, 1 healthy, 0 unhealthy
--> v1 deployed successfully

That's it! FreshRSS should be accessible from the URL <appname>.fly.dev. You can front a custom domain as well. Head over to the Certificates tab from Fly.io's dashboard, or use the flyctl tool to provision a certificate by typing flyctl certs add <domain>. Add the CNAME that Fly tells you about, and then add an A record for the (sub)domain, pointing to the Anycast IP you get from deploying FreshRSS. See Fly.io's docs on how to do this.

Caveats

FreshRSS extensions need to be available on /var/www/FreshRSS/extensions, however - Fly.io doesn't support having same mount for mulitple directories, or having multiple mounts.
- Trying to mount just /var/www/FreshRSS caused the microVM to throw kernel panics, mainly because the underlying Docker container expects something there - probably the webserver config and mounting an empty VM causes errors
- You could try to play with Fly's internal networking, including setting up a wireguard tunnel, but I didn't try.
- Another icky way is to build a custom Dockerfile copying the required extensions into the image during the build step..
I haven't run this for long, will update if I run into issues

Automated updates and deploys

Fly.io's CLI-oriented approach makes updates via GitHub actions easy. While I haven't set up for FreshRSS, you can look at this PR where I add a GitHub action to deploy my blog, sathyabh.at, a static site powered by Hugo to Fly.io.

Fly.io's community Discourse is quite active if you run into any problems. Hope this gives you a good taste of what Fly.io can offer.

See also: Github repo of the fly.io config file.

Second Edition of Practical Docker With Python is now available

Sathyajith Bhat — Mon, 20 Dec 2021 10:00:00 +0000

About 3 years ago, I wrote my first book - "Practical Docker with Python". The book focused on the fundamentals of containerization and getting started with Docker. A few months back, I received an email from my editor at Apress asking me if I'd be interested in doing a second edition of the book.

Looking back at the book, it had received good feedback, has been cited in different publications and there was some feedback I had received that I always wished I could update the book with, so this was the best chance. After many months of work, the second edition of Practical Docker With Python is available on SpringerLink, Amazon (US, India), O'Reilly Learning (formerly Safari Books Online) and probably every other online bookstore.

The second edition, much like the first edition, targets people who are new to containerization and want a guided approach to containerizing their application. I start with setting up a Python telegram bot, building and running it as a program, and then continue containerizing the same bot, starting with steps to build the Dockerfile, adding volumes for persisting the data, setting up Docker networks for container networking, and finally multi-container orchestration with Docker Compose.

What's new in this edition is a blurb on getting started with Windows Subsystem for Linux 2(WSL2), an entire chapter dedicated to Continuous Integration, and how a container image becomes from being a local thing to being deployed on an orchestrator like Kubernetes. Writing about Kubernetes was the hardest thing - entire books have been written on it and I certainly didn't want to focus on it, but I hope it gives a taste of what you can do with k8s.

Among the things that were left out include "docker compose", and multi-arch builds using buildx. The next gen docker compose (note: NOT "docker-compose"!) was just too new to be included in the book. But there is a reference to compose spec, and "docker compose" is supposed to be a drop-in replacement, so hopefully, you can swap out the commands. Not including docker compose meant I couldn't include deploying to ECS/other container orchestrators using docker compose.

I hope the second edition is a value add and you like it as much as I enjoyed writing it. Feedback, of course, is always welcome. My Twitter DMs are always open, or you can reach out to me on the channels mentioned in the contact page.

Checking for connectivity of ElastiCache Redis instances across peering connections using VPC Reachability Analyzer

Sathyajith Bhat — Mon, 13 Sep 2021 00:00:00 +0000

It's quite common to work in an environment where you have your VPC connected to multiple other VPCs, either via VPC Peering or Transit Gateway, it's no different for me. I was working on deploying an application on our in-house container platform. This container platform is hosted in another account/VPC, and it had to connect to the ElastiCache Redis instances hosted in our VPC/account. I was looking into timeouts connecting to the Redis instance from the application containers, and I figured this would be a good time to test out the VPC Reachability Analyzer.

The VPC Reachability Analyzer tool is a network diagnostic tool from AWS. You can access it from the VPC Section of AWS Console. Clicking on Reachability Analyzer, you enter the source, destination, optional intermediaries and click on "Analyze Path" for AWS to trace the path between the source and the destination.

I wanted to test the connection from the peered connection to the ElastiCache in the private subnet. I noticed the drop-down for destination (and the source ) did not have individual AWS services listed. That threw me off, as I couldn't select ElastiCache there, but Ben Bridts pointed out that each of the ElastiCache nodes would have an ENI attached, so providing the ENI as the destination should work. That said, finding the ENI was not the most straightforward option - I had to head to ElastiCache, pick up the URL of a node, do a dig on node DNS to find the IP address, and search for the IP address on the ENI page on the AWS Console (the Elastic Network Interfaces is shown as an EC2 feature, to make you even more confused). I think there might be avenues to simplify this workflow here.

I created some test paths to check how it works, and the experience was mostly positive.

For the first test, I created a path analysis with the peering connection as the source and the ENI of the ElastiCache Redis as the destination. However, I accidentally selected the wrong peering connection. When the test was completed, it showed a failure, and I was pleasantly surprised by the detailed error message, which outlined why the test failed:

For the next attempt, I tried to analyze the path between the correct peering connection and the Redis cluster, and the path trace failed - and yet again, the feedback and error message was pretty detailed in indicating where it was going wrong. It showed the two places where the connection was failing:

The route table didn't have the correct entries to route traffic from the peering into the private subnet.
The security group rules were incorrectly configured and were not allowing connections from the peering connection to the Redis port.

Clicking on the details shows the full info about which route table, which security group is preventing the network communication from happening.

Once I corrected the errors, the reachability analyzer said all was good!

I was still seeing the timeout errors from the application container. And therein lies the trouble with the VPC Reachability Analyzer - if you have multiple routes to a destination, the analyzer seems to consider the fastest path and ignores the failing routes. This is shown briefly when you create an analysis, and it mentions using a particular intermediate component filter to analyze the alternate paths. The intermediate components can be load balancers, NAT gateways, and peering connections but not security groups, ACLs, network interfaces or route tables. In this particular case, there doesn't seem to be a way to show alternate paths. I wish the Reachability Analyzer could use different subnets as an intermediate component filter. This should show the alternate paths where there are multiple routes via different subnets in different availability zones would show all the applicable paths and which ones are failing.

VPC Reachability Analyzer is still a fantastic tool to debug network connectivity issues, and I can see myself using this often. Path Analysis is charged at the rate of $0.10 per path analysis. And the fact that is done without sending any network traffic along the path is pretty awesome. For more details, you can refer to this re:Invent talk for details on hw AWS does this using automated reasoning.

Migrating my WordPress blogs to Hugo

Sathyajith Bhat — Fri, 28 Aug 2020 00:00:00 +0000

I started blogging with WordPress about 13 years ago. I had some free time since the joining date for my first job was about a month or so away. Armed with boredom, an Internet connection and an ample amount of free time, I started Sathya Says on WordPress.com hosting. Soon after, I came to know about domains, shared hosting and self-hosted WordPress and with my first ever salary, purchased sathyasays.com, shared hosting and started writing about Linux experiences. The shared hosting served me quite well, enough to get tens of thousands of views per month(which surprised me quite a lot since I focused on Desktop Linux, a tiny niche). I experimented with AdSense and managed to hit the $100 threshold in about 4 months. I removed Adsense soon after, there was just no point with such abysmal CTRs.

Fast forward a few years later, I was spending more time on keeping my WordPress install secure. The tipping point was when my DigitalOcean droplet was disconnected from their network as it was detected to be a contributor to a DDoS attack and it took DigitalOcean more than 2 days to re-enable networking so I can recover the data and fix it! I took this opportunity to disable WordPress and start exporting the data and import it into Hugo, to be hosted with Netlify. Here's how I went about exporting blogs.

Hugo

Hugo is a static site generator. Unlike other systems like WordPress which dynamically builds a response page on visit, Hugo builds the pages when the content is created/updated. This doesn't mean you'll have to painstakingly build HTML for each page - Hugo provides a framework for building the pages. Your pages will be in the form of a Markdown file which stores the content of the posts. Hugo also supports Frontmatter which lets you keep the metadata of the page, within the page itself. When you build a page in Hugo, it will parse through the Markdown file and generate a static HTML file. This static HTML can be hosted anywhere - from Amazon S3, Netlify etc. There's a lot of documentation on how to get started with but not a lot on how to migrate - so I thought I'll post on how I did the migration.

So how do we export the data from WordPress?

Before starting the migration, I had two major goals:

Data export from WordPress and import to Hugo should be done with as little work as possible
There should be no broken links - in other words, the permalinks, tags, category links etc should not change

While WordPress comes with a Data export tool, its an XML dump of all the content and I'd have to spend a lot of time in importing it. The Tools page on Hugo's website listed some tools for working with WordPress XML backup but didn't try them out.

With these two goals in mind, I tried out decided to use SchumacherFM's wordpress-to-hugo exporter. The exporter is actually a WordPress plugin - and comes in two forms - as a CLI as well as a proper plugin that you can activate from the WordPress plugins page. I would recommend using the CLI approach - the data export is quite intensive and you might end up with a 504/Time out/failed export if the export doesn't get completed within the timeout settings.

Step by step process

Below are the step-by-step process to export the data and import to Hugo

Clone the wordpress-to-hugo exporter repo on the server where your WordPress instance is running, in the wp-content/plugins directory
```
git clone https://github.com/SchumacherFM/wordpress-to-hugo-exporter.git
```
Change to the wordpress-to-hugo-exporter directory within
```
cd wordpress-to-hugo-exporter
```
Run the exporter
```
php hugo-export-cli.php
```

Depending on the size of your data and your server specifications, it might take anywhere from few seconds to a few minutes for the export to complete and will display the path where the file is saved.

hugo-export-cli.php
[INFO] tmp folder not found, use default. You could invoke php hugo-export-cli.php with an extra argument as the temporary folder path if needful.
This is your file!
/tmp/user/0/wp-hugo.zip

Download the file from the server using scp

scp user@remotehost:/path/to/wp-hugo.zip .

The compressed file will contain your blog data - posts, pages, images with the metadata in a structure and the config.yaml file that Hugo expects. Just download a Hugo theme of your choice and update the config file accordingly.

You can take a look at my blog repos for examples on structuring the directory:

Some tips and points to note

While the export is almost seamless, there are some pointers before you start the export to get the cleanest export:

Disable all your plugins, especially Jetpack. The plugins can often mess up the data exports with unnecessary cruft, making the Markdown exports bulkier than needed
The wordpress-to-hugo-exporter keeps any HTML content within the page(for example any embedded content, image galleries etc) as inline HTML. This wasn't a problem earlier but as of Hugo version 0.60, the default Markdown rendering library by Hugo, Goldmark, no longer renders inline HTML out of the box and you will have to enable unsafe mode as below:

config.yaml

  markup:
    goldmark:
      renderer:
        unsafe: true

config.toml

  [markup]
    [markup.goldmark]
      [markup.goldmark.renderer]
        unsafe = true

config.json

  {
    "markup": {
        "goldmark": {
          "renderer": {
              "unsafe": true
          }
        }
    }
  }

If you don't add these settings, the inline HTML would be omitted and replaced by this message

  <!-- raw HTML omitted -->

With the plugin, special characters may be exported as HTML entities and may not be rendered correctly by Hugo, especially in tags.

Hosting and Deploys

For hosting the imported blogs, I used Netlify. Netlify can connect with your Github repo and deploy the changes the moment a commit has been pushed, making it quite easy to write, review, edit and rework on the posts if needed. Netlify's deploy preview means that you can write a blog post, cut a Pull Request and Netlify will automatically publish a preview version - making editing and peer reviews easy, typically as comments on the Pull Request itself. Netlify's doing well as of now but have gradually started introducing more constraints and it is quite possible that Netlify might not be feasible in the near future, in which case I might just move the entire blog to GitHub pages, as described by Shantanu Goel in his post here.

Handling images

While using Git as the backend store for text was ideal, I wasn't happy with storing the blog images in the repo. I thought of using Git LFS and Netlify Large Media uses this, I ultimately didn't end up using it since it's metered under Netlify billing and Netlify's billing is not very granular. Ultimately I setup a Cloudfront CDN infront of an S3 bucket and started serving images off that. You can check Bhuvana's blog post on how to do this, or refer to my Github repo for the CDK code that I used to build the entire infra.

Commenting

For comments, there are a bunch of open-source self-hosted solutions - Utterances looks like the coolest of the lot, with comments powered by Github issues. Utterances requires a bit of theme hacking, so I didn't enable it. For now, I stuck to my existing Disqus account and might consider revisiting this later.

Running Folding@Home on AWS with AWS CDK

Sathyajith Bhat — Sun, 26 Apr 2020 06:16:24 +0000

Folding@Home(aka FAH) is a distributed computing project. To quote from their website,

FAH is a distributed computing project for simulating protein dynamics, including the process of protein folding and the movements of proteins implicated in a variety of diseases. Folding@Home involves you donating your spare computing power by running a small client on your computer. The client then contacts the Folding@Home Work Assignment server, gets some workunits and runs them, You can choose to have it run when only when your system is idle, or have it run all the time.

While I used to run FAH long, long back - dating back to my forum days, I eventually stopped due to lack of proper computing equipment. Recent events with the COVID-19 situation and FAH's projects around it (see Coronavirus - What we're doing and COVID-19 Small Molecule Screening Simulation for details) and the relatively powerful computer I built recently meant that I could run FAH on my desktop computer.

Now, I had some extra credits for AWS that were to expire soon and I figured instead of letting them go to waste, I thought to myself maybe I could spin up some EC2 instances and run Folding@Home on them. I started looking at the pricing of the GPU instances - and they were a bit pricier than what I could sustain. Considering this, I selected the c5n.large instance as I didn't need instance and EBS-backed disks would be handy in setting up aa Auto Scaling Group.

To reduce expenses further, I started looking at Spot prices and it turned out, the spot prices were about 68% cheaper as compared to the on-demand prices. Since we don't really care about what happens when the spot termination happens and the ASG will bring the instance count back up, I went with this option.

The spot pricing trend revealed that the prices had remained stable and just to ensure the spot bids would be fulfilled, I kept the max spot price couple of cents more than the maximum price going then. Initially, the instances were brought up by manually launching them from the AWS Console. Since long I'd been meaning to use AWS CDK, this was the perfect opportunity to learn and try to use it.

The CDK code will bring up a new VPC, a couple of subnets, an ASG and attach a security group to allow for SSH into the instance. The code is not the best, there's a bunch of hard-coding of regions, AMIs, SSH key names, but pull requests to clean up and make it more generic is more than welcome! Check out the code on my GitHub Repo

SathyaBhat / folding-aws

Bring up a complete AWS Compute stack with VPC, EC2, and other dependencies using AWS CDK. Set up a Folding @ Home stack with couple of commands

Folding on AWS

This is a CDK project which configures a multi-instance ASG. As an example, there are two sets of configs predefined: the first config creates a two-node ASG pointed to an AMI which is pre-configured to run Folding@Home, while the second config is for a single-node ASG running a base install of Ubuntu with some extras (check packer/generic_base cvonfig for details)

The AMIs are configured and built using HashiCorp's Packer. These AMIs can then be updated in the config file.

How to run

Preparing the AMI

Install packer
Generate AWS access keys
Set the following variables in your shell's environment: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION
Change into the packer sub directory: cd packer

Build the Folding at Home Amazon Machine Image that will be used to create the virtual machines

packer build -var 'fah_user=your_username' -var 'fah_passkey=your_passkey' \
            -var 'fah_team=your_team_id' fah_ami.json

If you have the jq program…

View on GitHub