DEV Community: Dev

samwise-CLI: The Open Source Terraform module dependency tracker

Dev — Mon, 12 Aug 2024 10:06:30 +0000

Terraform

Terraform is an Infrastructure-as-Code(IaC) tool that is written in Hashicorp Configuration Language(HCL). This article assumes that the reader has worked with Terraform and understand how modules work.

Everyone coding in Terraform has either created their own modules, or at least used someone else's.

Terraform modules

Modules are containers for multiple resources that are used together. A module consists of a collection of .tf and/or .tf.json files kept together in a directory.

Modules are the main way to package and reuse resource configurations with Terraform.

Modules can be published to registries like Terraform registry or GitLab. Modules can also be served from Git repositories by providing the HTTPS or SSH URLs. These can be versioned as well. For example,

module "consul" {
  source = "github.com/hashicorp/example"
}

Versioned:

module "consul" {
  source = "github.com/hashicorp/example?ref=1.2.6"
}

Challenge

As your repositories grow and you reference your modules in other repositories, you would reasonably version your modules to ensure that upstream changes in the source doesn't break your infrastructure. However, it is difficult to keep track of all the new releases for the modules being used and even harder to do it regularly. Unaddressed, this builds overtime as tech debt as one day you discover that a core module is now 3 major versions ahead.

Solution

Presenting samwise-cli, a tool to help track your repository's Terraform/OpenTofu dependencies upstream. Searches your repository for usages of modules and generates a report of the modules that have updates available along with all the versions that are more advanced than the version used currently.

At the moment, there's only one command, but hopefully there'll be more soon as the tool develops.

Darth-Tech / samwise-cli

A CLI application to accompany on your terraform module journey and sharing your burden of module dependency updates, just as one brave Hobbit helped Frodo carry his :)

samwise

A CLI application to accompany on your terraform module journey and sharing your burden of module dependency updates, just as one brave Hobbit helped Frodo carry his :)

                       \ : /
                    '-: __ :-'
                    -:  )(_ :--
                    -' |r-_i'-
            ,sSSSSs,   (2-,7
            sS';:'`Ss   )-j
           ;K e (e s7  /  (
            S, ''  SJ (  ;/
            sL_~~_;(S_)  _7
|,          'J)_.-' />'-' `Z
j J         /-;-A'-'|'--'-j\
 L L        )  |/   :    /  \
  \ \       | | |    '._.'|  L
   \ \      | | |       | \  J
    \ \    _/ | |       |  ',|
     \ L.,' | | |       |   |/
    _;-r-<_.| \=\    __.;  _/
      {_}"  L-'  '--'   / /|
            |   ,      |  \|
            |   |      |   ")
            L   ;|     |   /|
           /|    ;     |  / |
          | |    ;     |  ) |
         |  |    ;|    | /  |
         | ;|    ||    | |  |
         L-'|____||    )/   |
             % %/ '-,-

…

View on GitHub

checkForUpdates

For this to run where modules are using private GitHub repositories, .samwise.yaml config file needs to passed as an argument or needs to be present at the user's home directory.

.samwise.yaml format:

git_key:
git_username:
git_ssh_key_path:

or as environment variables:

SAMWISE_CLI_GIT_KEY
SAMWISE_CLI_GIT_USERNAME
SAMWISE_CLI_GIT_SSH_KEY_PATH

Result

CSV Format

Limitations(or better described as features to be added)

~~SSH authentication for retrieving module sources~~(support added)
Tracking Hashicorp's registry's module(they have an API to list versions here)
Auto-creation of PRs with the updates(added experimentally with only committing support, no push and PR yet)

Context for the name

I love Lord of the Rings :)

Journey Through DevOps - Part 3: Higher

Dev — Sat, 21 Aug 2021 10:51:45 +0000

This is 3 part series documenting my journey through Software development. For Journey Through DevOps - Part 2: The Awakening, click here

This post will be an insight into the automation that we use in our projects. This is not a tutorial but a demonstration of how automation can change the software process.

Background

We manage the deployment of 2 applications - Chatwoot and Rasa-X. We aim to deliver content through the Natural Language Processing framework of Rasa and use Chatwoot to track and intervene in the chat, if necessary.

Technology stack

Cloud: AWS
Deployment: Helm, Kubernetes(1.21, Elastic Kubernetes Service)
Datastore: Postgres, Redis (AWS managed)
Applications: Rasa-X(Python), Chatwoot(Ruby on Rails)
VCS: Gitlab

Setup

Kubernetes

We use AWS EKS for deploying our applications. The pods run on both FarGate and Managed Node-groups to get the best of both worlds. Since the micro-services of chatwoot are stateless we use FarGate, whereas the Rasa-X deployment uses node groups for certain pods that need fewer resources to run and also need to be stateful.

Infrastructure

For AWS infrastructure, we use Terraform. The repo is connected as a Version control system to Terraform cloud allowing us to focus more on the infrastructure itself instead of having to CI pipelines.

Helm

We use Helm charts for both Rasa-X and Chatwoot, latter of which was built by us. Helm allows us to have a standard way to deploy the application instead of a lot of YAML files which need modification every time there's an update.

The Automation

Automation by its nature, is intuitive. General rule of thumb that we follow, is that if a particular task is done more than 3 times, it is best to automate it.

Let's start with the lightweight components.

Chatwoot

Since chatwoot is an open source project and we don't add a lot of custom code to chatwoot, there isn't any need for CI/CD pipelines for chatwoot. Every update, it's easier to run

helm upgrade <release-name> chatwoot/chatwoot

Infrastructure

Since we're using Terraform cloud, this is also very simple. Although we did consider using Terraform locally in pipelines, this was an overhead considering as our team was small. Note that you can use Terraform cloud for both execution of Terraform and storing the Terraform state alone.

Rasa-X

This is the component that involves the maximum number of repetitions, and therefore is fully automated.

Rasa-X has several components but we'll focus on the important ones for now, such as Rasa open source and Rasa Custom Actions server. Former is the NLP part of Rasa and the latter is a python server that lets you run custom events in your chatbot such as fetching data from a database. Since chatbots are iterative, it was imperative that we automate this process before moving ahead.

CI/CD

This part requires you to repetitively train NLP models and test them just as much. Hence use gitlab-ci pipelines to train a model, subsequently test it and upload the results as artefacts.


build-actions:
  image: docker:20.10.7
  stage: build
  services:
    - docker:20.10.7-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - chmod +x ./ci.sh
    - ./ci.sh

train-job:       
  stage: train
  script:
    - rasa train --fixed-model-name $CI_COMMIT_SHORT_SHA
  artifacts:
    paths:
      - models/
    expire_in: 1 day

data-validate-job:  
  stage: validate    
  script:
    - rasa data validate

core-test-job:   
  stage: test    
  script:
    - rasa test core --model models/ --stories test/ --out results

nlu-test-job:   
  stage: test    
  script:
    - rasa test nlu --nlu data/nlu.yml --cross-validation

upload-job:
  stage: report
  script:
    - echo "Upload results"
  artifacts:
    paths:
      - results/

This can be broken into 2 parts. The custom actions server part and the Rasa bot.
Custom actions is a python server, so we dockerize to allow for easier development and standardisation. The "./ci.sh" is a shell script to read the branch name from the environment variable and set the docker image tag accordingly, to build and push it to the registry.

The next parts of this pipeline, train a model, validate that model for any conflicts and errors and finally run tests on them. All the tests are uploaded as artefacts in the pipeline. This allows us to log every model and iteration. The --fixed-model-name tag is to ensure that models have a predictable name which can be used further in the pipeline. Rasa defaults to using Timestamp which can be unpredictable.

Branches:

main: This is the branch that represents production, so every piece of code that resides here is battle tested and verified. Hence it is safe to assume that with every push into this branch, we can update the NLP model in the server.

Docker tag for custom actions: stable

deploy-job: 
  stage: deploy 
  #before_script: []
  #image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest  # see the note below
  script:
    - apt-get update
    #- apt install git-all -y
    - curl -k -F "model=@models/$CI_COMMIT_SHORT_SHA.tar.gz" "http://rasax-url.com/api/projects/default/models?api_token=$RASAXTOKEN"
    - echo "Application successfully deployed."
    - aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME}
    - curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
    - helm plugin install https://github.com/rimusz/helm-tiller --kubeconfig=$HOME/.kube/kubeconfig
    - helm repo add rasa-x https://rasahq.github.io/rasa-x-helm  
    - helm upgrade rasa rasa-x/rasa-x -n rasa --set app.name=$CI_REGISTRY/weunlearn/wulu2.0/rasa_actions --set "app.tag=stable" --reuse-values   # Redeploys the kubernetes deployment with a new image name while reusing already existing values      

  rules:
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "main"'

This stage in gitlab pipelines that is only executed when the event is a push event and the branch committed to is main. This prevents any unnecessary triggering of model training for minute changes in various other branches.

develop: Since this is a non-production environment, there is no deployment from this branch. The deploy stage from the previous section is not executed. However, since this branch serves as a reference point before updating production code in the main branch, it has a static name.

The test results are uploaded as before. This allows us to keep track of every model iteration.

Docker tag for custom actions: develop

Workflow

Pushes into non-main, branches:

Pushes into main

The pipeline is the same as before except it has a final stage

Conclusion

This setup enables a tech team of 2 people, to manage multiple applications in our production environment, while also ensuring a standardised way to perform quality control. Now the goal of the tech team is to solve problems and build the product, as opposed to dedicating a significant share of time to figuring out how to manage the product.

Journey Through DevOps - Part 2: The Awakening

Dev — Sun, 20 Jun 2021 16:36:54 +0000

This is 3 part series documenting my journey through Software development. For Journey Through DevOps - Part 1: World without DevOps, click here

After a turbulent initiation into software development, one thing was certain, I was a total novice and had to up my game. It was at this junction that I'd taken my first Cloud computing course, AWS Fundamentals: Going Cloud-Native. Now, this course taught me multiple concepts such as Failover, Availability zones and Load Balancers.

But there's were a few issues. As the complexity of the infrastructure grew, with it grew:

Necessity to manage the infrastructure.
Need for developing a software development methodology that would allow changes in applications to be easily integrated with existing infrastructure.

The former issue was reflected in the event where I had lost track of an AWS Global Accelerator(networking service that improves the performance of your users’ traffic) and it went on to cost us credits for a few months. I had manually search for the resource across all regions, which is less than ideal. The latter issue was reflected when there were multiple Natural Language Processing models and they had to be manually updated in multiple instances.

Enter DevOps

My dad is a software engineer so I'd always hear words and technologies that they were using in their workflow. And upon looking up said words, I'd find out they solve a problem that I am facing in my workflow. Although explaining DevOps is something that beyond the scope of both me and a blogpost. These are the tools on my journey that have enabled me to solve problems and implement practices of DevOps.

CI/CD(Problem 2)

Jenkins

It's a versatile tool to execute both Continuous integration and Continuous Delivery(CI/CD). Jenkins is a server based tool which can define CI/CD processes as Pipeline as a Code. So that means there's an extra overhead in installing Jenkins master node and agent nodes. So for a small scale project as ours, its versatility was the bane.

We needed something lightweight.

GitHub Actions

GitHub Actions is a relatively new feature in GitHub where we can declare build and test processes in GitHub as YAML files. These are then executed either on GitHub runners(basically virtual machines hosted by GitHub) or self hosted runners. These are a game changer because they allow many simple processes to be executed simply by pushing code or creating a pull request.

However, they only solve half the puzzle. GitHub actions are extremely easy to incorporate with CI processes however, the CD part remains a problem. Apart from basic support to Kubernetes clusters(Only cloud provided) there are very few CD options.

However until our second release, GitHub actions sufficed, so we used them.

Hashicorp(Problem 1)

After our first project, it was apparent we were scaling up infrastructure wise. Enter, Terraform

Terraform

My love affair with Hashicorp began with Terraform. In all honesty there may not have been that desperate a need for it, I used it because it seemed cool, although it is impossible to separate Terraform from my current role.

To check out more about Terraform and its uses checkout my post on medium.

My misconception about Terraform was that it was to deploy infrastructure like we would to an application. This would mean I write code for the infrastructure needed and I would deploy it on Day 0, and then install my application on relevant resources. However, that turned out to not be feasible. Terraform is a tool that is a part of your daily workflow. Incremental changes to infrastructure are logged and created by Terraform allowing you to track and visualise infrastructure you are using and how it fits together, like a jigsaw puzzle.

Although there are tools such as Pulumi which offer IaC(Infrastructure as Code) in programming languages like Python or Node, I prefer Terraform because of it's simplicity and ease of use.

Conclusion

Granted these were major leaps from the previous experience(i.e described in previous post), the point of writing these posts to hopefully let people know that what you build at first will be horrible and you will cringe at it when you look back. However as one of my favourite movies said about cooking, we can safely bet "Not everyone can be a great developer, but a great developer can come from anywhere."

Kubernetes on Raspberry Pi

Dev — Tue, 01 Jun 2021 19:33:38 +0000

My friend had developed a proxy server in PHP as a part of his semester project. We both discussed the project and realised that this simple project gave us an opportunity to learn and apply real world concepts as opposed to letting a wonderful proxy server decay as just a proxy for grades.

Enter, Kubernetes.

I had been personally looking up Kubernetes to understand and hopefully experiment with it for a while, however without an application to deploy, my theory was ineffective. So we decided to deploy my friend's Proxy Server application on his Raspberry Pi. This allowed us to simulate the experience of accessing an actual cluster as opposed to Minikube which would be dependent on our machines being up all the time.

It wasn't complicated at all. We dockerised and tested the application on RaspberryPi and Docker. Following which the Kubernetes deployment was a walk in the park.

Read more about the application and deployment here

Thank you!

Journey through DevOps - Part 1: World without DevOps

Dev — Tue, 01 Jun 2021 19:20:26 +0000

April 2020. I'd just begun coding regularly in Python. Built a toxic tweet classifier using LSTM models. Model wasn't spectacular by any standard, but during the whole process, I was introduced to something that would change the way I look at software development completely. It was, the Cloud.

I need to run the model faster so I used Virtual Machine in Google Cloud Platform to run the training on a higher configuration. And that was about it.

Fast forward a few months. August 2020. I was experimenting with random tools of NLP and tested out a chatbot using Rasa. Because of this minor experience, I was connected by a friend to an NGO, Weunlearn which had proposed to build a chatbot for providing Gender, Sex-ed and Mental health content. Needless to say, I signed up to help. Because for me, a chatbot simply meant a bunch of NLU training data and conversation stories.

After basic onboarding, it became quickly apparent that I was woefully under-prepared and under-equipped because we were soon to have a bot but where would it go?

Soon, the NGO had obtained credits in AWS through the Activate Program. Now it was my responsibility to put them to good use.

And I did. Just in a clumsy, haphazard and unprofessional manner.

In the first iteration of the chatbot, technically we faced a lot of complications that I feel require documentation. I believe we should share our stories not as simple tech tutorials or guides but also as journeys because it is part of what separates us from the machines we so passionately work on. What follows are the mistakes, glitches and poor planning decisions that only a software noob would do.

1. GitHub Chaos

Let's start with in reverse chronological order of hiccups. Git is a version control system(VCS). GitHub is a cloud based service that allows you to host your git repositories. GitHub(or any VCS) allows for easy collaboration and code management across the team. The tech team at the NGO was 2 people, including me and we chose GitHub to store the repos.

1.1 First mistake - Binaries in GitHub

GitHub is a place to store code files. Not binaries. Without understanding this, we had stored the Rasa NLP models in the GitHub repo. Which was fine until the day of deployment. A few last minute increases to NLP training data had pushed the model above the GitHub file limit. 2 hours to deployment and first message.

Needing to find alternate methods to store the binary and deploy, we turned to...

1.2 Second - Git-LFS

Needless to say, as a complete beginner I was unaware of how git-lfs works. I chose to use it in the heat of the moment. Git-LFS works by replacing the file contents with a pointer. If one of the system using the repo does not have git-lfs, the whole thing breaks down. And it did.

I was left with multiple Dockerfiles with no docker related lines but a pointer to a git lfs object we could not reach. What happened after that is a hazy mess as best, but we did manage to SCP the code from a local machine to the VM.

Lessons:

Always learn how to use VCS. Binaries do not go into GitHub. Use storage services like S3(free tier) or Google Cloud Storage(provides a 300$ starting credit for new sign up). If you do want to use a tool, understand the consequences before going ahead.
Don't experiment on the day of deployment

2. Deployment Unstrategy

I had abundant credits in AWS and fair knowledge of Docker. So we decided to use a simple EC2 instance to host the application.

Rasa provides minimum system requirements to run their bots and we adhered to that. However, we were unsure of what to do in cases of overload. Although we tested with only a handful of users, inexperience got the best of us.

Failover strategy: To download the code into another VM(in Stopped state) and wait for the application to fail, so that we can manually start the backup instance.

However, I'd not setup a monitoring system to alert anyone for failures. Which essentially meant we had no way of knowing if it failed. It also meant any changes in training data would require manually updating multiple instances and verifying them.

Lessons:

Load testing is an important tool in your arsenal k6 in javascript or locust in python are wonderful tools.
Always try to ensure that your operations overhead is less(more on this in next topic). Use cloud based infrastructure to monitor your application as much as possible.

3. CI/CD

Continuous Integration/Continuous Delivery is one of the most important practices that a developer would need as they are moving from just coding to developing applications. Application development doesn't always mean cloud and even if you dont have access to cloud resources, CI/CD is something that should be part of your day to day development.

Continuous Integration means whatever changes you make in our code, are easily added to the source code or it's binaries. For example, if your application is executed as a binary, then every time you update the code, the latest binary must be built on its own. This allows you to focus on the code part of it, than building versions of binaries. Git uses branches to version code, any update in master branch may build a stable binary, whereas in the development branch the binary maybe tagged "latest". Allowing for easier understanding of changes and versions of code.

Continuous Delivery is taking the updated artefacts(binaries, docker images) and updating the application servers without manual intervention. Let's say you use Docker images in your application. Every push into the master branch should trigger a CI workflow which build a docker image and stores it securely. CD workflow would replace the outdated docker image in the application and restart application with the new one.

All done automatically.

None of this was a part of our first iteration of the bot. Models were manually built, testing was unheard of and any changes had to manually modified in each server.

Lessons:

Even the smallest of the application benefits from CI/CD practices.
It helps ease workload by automating most of it

As is evident from these, our bot was shabby and clunky technology wise. It made our job a lot harder because we had to deal with problems that arose not from the product, but from the practices that were being followed.

The story however, does have a happy ending. Watch out for Journey through DevOps - Part 2: The Awakening.