DEV Community: Mike Mwanje

Configure a MongoDB cluster with Ansible

Mike Mwanje — Sat, 13 Jan 2024 08:39:00 +0000

Manual server configuration is always a nightmare, especially when there is a large fleet of servers to manage. However, with Infrastructure as Code(IaC) automation tools like Ansible, all that pain goes away. In this blog, we shall leverage the automation tool to create a MongoDB sharded cluster from scratch in just a few minutes, using this Ansible role.

Prerequisites

Ansible installed on your control node
Seven (or more) target servers running any Ubuntu OS version
SSH access to the target servers

Getting our hands dirty

Install the Ansible Role

As I mentioned earlier, we shall be using an existing Ansible role and so we won't have to write all the playbooks by ourselves from scratch. However, to install the role, we shall need a file that has it's installation information. A requirements.yml file . Run the commands below to create the file and the directory, ansible_mongodb, which we shall be working.

mkdir ~/ansible_mongodb
echo "-   name: 123mwanjemike.ansible_mongodb
    src: https://github.com/123MwanjeMike/ansible-mongodb
    version: v1.0.4" > ~/ansible_mongodb/requirements.yml

Note: The latest version of the role, 123mwanjemike.ansible_mongodb, at the time of writing this blog, was v1.0.4. There may however be more recent release versions at the time you read this which are more recommended to use instead.

You can now install the role using the Ansible Galaxy command below.

ansible-galaxy install -r ~/ansible_mongodb/requirements.yml

Output:

The Inventory File

Create an inventory using the command below

mkdir ~/ansible_mongodb/inventory
touch ~/ansible_mongodb/inventory/hosts

Next, populate the inventory with the target servers grouped according to the respective replicaSets. That is to say, the Config servers replicaSet servers in their group, and the Shard servers in the corresponding group.

Also, make sure to use the Fully Qualified Domain Names(FQDN) of the servers when adding them to the inventory. Below is how my inventory looks like.

all:
  children:            
    mongo_cluster:
      children:
        mongos_routers:
          hosts:
            mongos-router-0.europe-north1-b.c.oceanic-muse-408212.internal:
        config_servers:
          hosts:
            mongod-cfgsvr-0.europe-north1-b.c.oceanic-muse-408212.internal:
            mongod-cfgsvr-1.europe-north1-c.c.oceanic-muse-408212.internal:
            mongod-cfgsvr-2.europe-north1-a.c.oceanic-muse-408212.internal:
        shard_0:
          hosts:
            mongod-shard-0-0.europe-north1-b.c.oceanic-muse-408212.internal:
            mongod-shard-0-1.europe-north1-c.c.oceanic-muse-408212.internal:
            mongod-shard-0-2.europe-north1-a.c.oceanic-muse-408212.internal:

The Variables

There will be a number of variables at play when using this role. Let's get them right.

Host Variables

Run the commands below to put them in place

All hosts

mkdir ~/ansible_mongodb/inventory/group_vars
echo "ansible_python_interpreter: /usr/bin/python3" > ~/ansible_mongodb/inventory/group_vars/all.yml

Router(s)

echo "cluster_role: 'router'" > ~/ansible_mongodb/inventory/group_vars/mongos_routers.yml

Config Servers

echo "cluster_role: 'config'

replica_set:
  name: 'cfgsvr'
  group: 'config_servers'" > ~/ansible_mongodb/inventory/group_vars/config_servers.yml

Shard_0 Servers

echo "cluster_role: 'shard'

replica_set:
  name: 'shard-0'
  group: 'shard_0'" > ~/ansible_mongodb/inventory/group_vars/shard_0.yml

Configuration Variables

You can find the role's preset configuration variables in the var/main.yml file at the path where the role was installed. For example in my case, it was installed at ~/.ansible/roles/123mwanjemike.ansible_mongodb.

You can change them as you wish, especially the mongo_configs versions and public keys given that they point to Ubuntu Jammy's MongoDB version 7.0 public keys.

Cluster Variables

As for the MongoDB cluster configuration variables, they have been documented on the role's README on GitHub here.

A Playbook

We're almost good to go. We now just need a playbook to install and configure the MongoDB sharded cluster. I build from the sample playbook provided in the README of the role on GitHub. Below is my ~/ansible_mongodb/sample_playbook.yml file

- hosts: mongo_cluster
  become: true
  vars:
    adminUser: "myAdminUser"
    adminPass: "" # Secret password here
    new_shard:
      name: "shard-0"
      group: "shard_0"
    tgt_db: "myDatabase"
    roles: ["readWrite", "userAdmin"]
    userName: "targetUser"
    userPass: "" # Secret password here
    keyfile_src: "./keyfile"
    cfg_server:
      name: "cfgsvr"
      group: "config_servers"

  pre_tasks:
    - name: Generate random string with OpenSSL on ansible controller
      shell: openssl rand -base64 756 > keyfile
      delegate_to: localhost
      args:
        creates: keyfile

  roles:
    - { role: 123mwanjemike.ansible_mongodb, flags: ["install_mongo"] }
    - { role: 123mwanjemike.ansible_mongodb, flags: ["configure_mongo"] }
    - { role: 123mwanjemike.ansible_mongodb, flags: ["clear_logs"] }
    - { role: 123mwanjemike.ansible_mongodb, flags: ["prepare_members"] }
    - { role: 123mwanjemike.ansible_mongodb, flags: ["start_mongo"] }
    - {
        role: 123mwanjemike.ansible_mongodb,
        flags: ["init_replica"],
        when: cluster_role != 'router',
      }
    - {
        role: 123mwanjemike.ansible_mongodb,
        flags: ["create_admin"],
        when: cluster_role != 'router',
        delegate_to: "{{ groups[replica_set.group] | first }}",
      }
    - {
        role: 123mwanjemike.ansible_mongodb,
        flags: ["add_shard"],
        when: cluster_role == 'router',
        run_once: true,
      }
    - {
        role: 123mwanjemike.ansible_mongodb,
        flags: ["create_database"],
        when: cluster_role == 'router',
        run_once: true,
      }

The moment of truth has now come. We shall be running the playbook with the command below and see what happens.

cd ~/ansible_mongodb/
ansible-playbook sample_playbook.yml -i inventory/hosts

Output:

Checking out the cluster 🎉💃🏽🕺🏽😎

Closing Remarks

With this Ansible role, we've been able to create and configure a MongoDB sharded cluster in under 10 minutes without writing any single configuration for the servers by ourselves.

Incase you were wondering, I am indeed the author of the role we have just used to create the cluster. I am also working on this Terraform project that can be used to create the server instances for the sharded cluster used herein. Currently, I have only set up the provider for Google Cloud Platform(GCP) but will be adding Azure, AWS, and other service providers soon. I hope you enjoy them as much as I did when creating them.

Cheers!

Additional resources

My experience with GitOps. So far...

Mike Mwanje — Sat, 26 Nov 2022 12:57:10 +0000

GitOps is a modern way to manage cloud-native systems that are powered by Kubernetes. It leverages a policy-as-code approach to define and manage every layer of the modern application stack - infrastructure, networking, application code, and the GitOps pipeline itself.

-- Weaveworks

GitOps as a concept was established in 2017 by Weaveworks and has since been widely adopted in software delivery with a growing number of CNCF tools being developed around it.


Source: GitOps: High velocity CICD for Kubernetes

While at AirQo, I've had the opportunity of working in both traditional and GitOps environments and in this article, I share experience from a DevOps perspective after having made the shift in our delivery pipeline.

Some bit of context

AirQo is a cleantech startup whose vision is clean air for all African cities. A key duty in getting there is availing our users with Africa's air quality information and so we have a number software products ranging from mobile applications, web applications, and an API just for that. Our API relies on a couple of microservices in the backend to transform the Particulate Matter(PM) data collected by our air quality monitors into consumption-ready information. We run our microservices on some few Kubernetes(k8s) clusters.


Architectural diagram

We use a Gitflow worflow in our mono-repository with all the products and the corresponding k8s manifests in the same repo.

Initial setup

In the early days, our continuous delivery pipeline had both the integration and deployment done by the same tool, GitHub Actions. All application images pointed to the "latest" tag in within k8s configurations thus it was the de facto for all applications restarts. GitHub Actions built and pushed the images to Google Container Registry(GCR), but was did not update the corresponding k8s manifest with the new image tag despite the fact that it was the uniquely identifiable image tag(e.g prod-docs-123abc). This meant that the deployed state was always divergent from the state on GitHub at the image tag level at least. Some of the challenges we faced include;

Difficulty in telling the actual resource definition of applications on the cluster. There were times when quick/minor fixes were made to applications directly through kubectl without going through the git workflow. It was difficult for the rest of the team to know of such changes given that they were rarely communicated or committed to GitHub. Not to mention that the image tag was obviously one of such application properties we weren't sure about.
The development team lacked visibility into their applications. Most development team members weren't conversant with k8s and kubectl commands to for instance view live application logs whenever need arose. This made troubleshooting unnecessarily a step longer given since a DevOps engineer was needed to retrieve the logs and thus,
A larger workload for the operations team given that it was an operations-centric approach.
Operations engineers always had to have access to the clusters via a kubeconfig in order to carry out even very basic tasks.
Rolling back changes was more technical than necessary. Given that we defaulted to the latest tag on GitHub. A DevOps engineer had to get the image tag of a previous application version from GCR, then run a kubectl patch command to update the application image tag with a previous one.

How we got there

A mono-repository setup can present some challenges and we faced one ourselves with the different products' delivery pipelines to production getting intertwined; deeming it impossible to deploy a specific product at a time. The CI/CD pipelines were all dependent on the git workflow and thus had the same trigger. Perhaps I can discuss how we went about in another article but during my search for an alternative autonomous trigger, is when I landed on ArgoCD. Up until then, I had miniature knowledge of GitOps but I found it interesting as I read about it. Fascinated by how the methodology would address many challenges we faced, I instantly installed ArgoCD on one of our clusters. Due to it's multi-tenant nature, I was able to add setup applications in the different clusters and their respective projects under one roof: and subsequently on boarded the entire team onto it.
We now use ArgoCD as our deploy operator but kept GitHub Actions as our CI tool and Config Updater for image tags on Helm(which we defaulted to).

ArgoCD removed pain points

With ArgoCD, we are notified via slack whenever the state of the deployed application diverges from the state on GitHub. The states are also automatically synced to match the GitHub definition in line with the GitOps principle of Git as the source of truth. This ensures that changes are documented on GitHub as a bare minimum.
The development team is now able to self serve on anything to do with their applications. The ArgoCD UI is so intuitive that they can carry out tasks ranging from creating new applications to accessing logs of existing ones without assistance from the operations team. A more developer-centric approach.
Rollbacks are now much easier. It's now a matter of reverting changes on the GitHub repo and an older image tag will be deployed automatically.
As the DevOps engineer, I am now able to run basic operations on the clusters without using and scripting commands. For instance, I can view health statuses of all applications on the clusters at the comfort of my phone, without running any commands to switch cluster context.

Final words

Our adoption of GitOps is recent but I can definitely tell that there's a lot we have to benefit from it at AirQo as we continue to explore it. I thus generally believe that what I've shared here is only but the surface and trust that you and your organisation could discovered that this methodology solves challenges you had gotten accustomed when you try it.

Until next time... Happy hacking

Further reading

Creating a new website? I have Nuttertools for you.

Mike Mwanje — Sat, 03 Sep 2022 09:15:57 +0000

Ever played Grand Theft Auto? If you have, then you might be familiar with "Nuttertools". Nuttertools is one of the many cheat codes in the video game which gives your character a set of weapons. In this article, I'll share some cheat codes of my own with you. A special group of websites that will make you stand apart from any ordinary web developer. I call these my Nuttertools.

1. webpagetest.org

With this website, you get to run performance, lighthouse, core web vitals, visual comparison, and trace route tests on your website. It can simulate a wide range of internet connections right from 2G to cable in various locations around the world. The tests are recorded and so can can playback to see how your website gets displayed on the browser and also share the recorded results if you wish to.

2. validator.w3.org

This W3C markup validator checks for areas of improvement and errors in your website HTML syntax and suggests how these can be addressed. As you can see in the image above, I have some of my own I can fix.

3. wave.webaim.org

The internet has a very large number of users who are diverse in many of ways. This tool comes in handy if you wish to create a website that is accessible to a wide audience, including those with visual, audial, physical, and cognitive impairments. It helps you identify implementations that may not be favourable for impaired individuals and how these can be can corrected. An example is colour contrasts that may not be distinguishable by people with colour blindness as well as they would be for normal users. This way you can promote inclusivity even in your web solutions.

4. whatsmydns.net

Let's say you've gotten a new domain and/or made a couple of changes. These changes can take a some good time, up to 72hours to reflect worldwide on the all the Domain Name Servers. This website helps you check the DNS propagation.

5. websitecarbon.com

Last but most definitely not the least, is a tool that calculates how much carbon is released into the atmosphere whenever a user visits your website. You can then see how much your website contributes to global warming and also suggests how you can reduce the carbon footprint. If there's at least one take away from this article, I hope this be it.

Do you know of any other "Nuttertools" that aren't on this list, do share in the comments section. I'd love to learn from you too.

Cheers!

Set up Kubernetes role based access control

Mike Mwanje — Wed, 24 Aug 2022 20:03:07 +0000

Kubernetes(K8s) role-based access control is a powerful tool in restricting access to resources within a Kubernetes cluster. In this post, we shall briefly discuss what role-based access control is, and how to set it up in Kubernetes. I promise, it's not as long a process as you may think.

What is role-based access control?

Role-based access control(RBAC) is a security model in which users are assigned roles, and can then be granted access to certain resources based on those roles. For example, an organisation can have a role-based access control system that allows DevOps engineers to perform privileged operations within the entire Kubernetes cluster, while restricting the application developers to only viewing information about their applications deployed in a given namespace.

Definitions

Before setting up RBAC in Kubernetes, let's review some of the key concepts that we are going to be dealing with:

Service accounts are identities that can be used to delegate access to Kubernetes resources within a namespace. They can be used by users or services. We shall do some minor tweaks later on in the article using clusterrolebindings to enable us override the "namespacedness" limitation.
Resources are objects within a Kubernetes cluster that we aim to implement role-based access control around. Examples include pods, deployments, replicasets, configmaps, and services.
Roles are collections of permissions that are applied within a given namespace. In most cases, these consist of a set of permissions that can be granted to service accounts.
Rolebindings are the association of a role with a service account. In other words, rolebindings are used to assign roles. Think of a rolebinding as a mechanism to plug a set of permissions(a role) to a service account.
Clusterroles are roles that can be assigned to service accounts within a Kubernetes cluster. Whereas roles are namespaced, clusterroles apply to the entire cluster. Some default ones in a k8s cluster include view, edit, admin, and cluster-admin.
Clusterrolebindings are associations of a clusterrole with a service account. As you might have guessed, clusterrolebindings enable us assign cluster wide permissions to a service accounts through clusterroles.

Now let us see how we can set up RBAC in Kubernetes.

How to set up a Kubernetes cluster using RBAC

Prerequisites

A working Kubernetes cluster. If you don't have one yet, you might find this post helpful. It lists a couple of ways you could setup a K8s cluster.
Kubectl. You can install kubectl following this guide.
Kubernetes 1.20 or later. Earlier versions of Kubernetes are no longer actively maintained and RBAC in those versions is not as stable. This documentation can guide you through the process.
The Kubernetes Dashboard if you intend on using it in the Access the Kubernetes dashboard section later on.

Setting up RBAC within a namespace

First, let us create a namespace which we are going to be working with. Let's call it dev-env.

kubectl create namespace dev-env

Output:

Next, create a service account called app-dev which we shall use all through out.

kubectl create serviceaccount app-dev --namespace dev-env

Output:

We shall follow this up by creating an admin role that has all rights within the dev-env namespace. Below is its manifest, which is also available on GitHub.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: admin
  namespace: dev-env
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

Apply the file to create the admin role.

kubectl apply -f https://raw.githubusercontent.com/123MwanjeMike/k8s-rbac/main/admin-role.yaml

Output:

Note: We shall be creating the rest of the Kubernetes resources in this tutorial from the pre-written manifest files in this repository as we have done above. However, you may also edit these manifests if you wish to create K8s resources tailored to your needs.

123MwanjeMike / k8s-rbac

Setting up Kubernetes role based access control tutorial manifests

k8s-rbac

A list of Kubernetes resources for setting up of role-based access control in the article Set up Kubernetes role based access control

View on GitHub

Next,create a rolebinding, app-dev-rolebinding, of the admin role to the app-dev service account. Below is the manifest.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: app-dev-rolebinding
  namespace: dev-env
subjects:
- kind: ServiceAccount
  name: app-dev
  namespace: dev-env
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: admin

Run the command below to create it.

kubectl apply -f https://raw.githubusercontent.com/123MwanjeMike/k8s-rbac/main/rolebinding.yaml

Output:

Our service account is now an admin within the dev-env namespace. We can even go further and define access rights for it across the entire cluster. Let's say, basing on our DevOps engineer and application developer example we had at the start, that we want our application developers to view any resource cluster wide and even to use the Kubernetes dashboard. How we can we do this?

Setting up RBAC for the entire cluster

For the first part, we're in luck! Kubernetes has a default view clusterrole which can be used for viewing of all resources in a cluster; and so we shall just leverage this. We shall create a clusterrolebinding, app-dev-view, that does just that.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: app-dev-view
subjects:
- kind: ServiceAccount
  name: app-dev
  namespace: dev-env
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view

Create the clusterrolebinding

kubectl apply -f https://raw.githubusercontent.com/123MwanjeMike/k8s-rbac/main/clusterrolebindings/view.yaml

Now, for app-dev to access the Kubernetes dashboard, we could easily just create a bind the edit default clusterrole to it since it has the access rights needed for this. However, this would also give the service account more permissions than it should have and so we want to use a more fine-grained clusterrole in accordance to the principle of least privileges.

So, we shall create a custom clusterrole with only the required permissions needed to access the K8s dashboard on top of those already held by the service account through default view clusterrole.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: view-addition-for-k8s-dashboard-access
rules:
- apiGroups: [""]
  resources: ["pods/attach", "pods/exec", "pods/portforward", "pods/proxy", "services/proxy"]
  verbs: ["get", "list", "watch", "create"]
- apiGroups: [""]
  resources: ["services"]
  verbs: ["proxy"]

Run the command below to create the view-addition-for-k8s-dashboard-access clusterrole with the above definition.

kubectl apply -f https://raw.githubusercontent.com/123MwanjeMike/k8s-rbac/main/custom-clusterrole.yaml

We can now bind view-addition-for-k8s-dashboard-access clusterrole to the app-dev service account for it access the K8s dashboard. Let's have a look at the clusterrolebinding manifest for this.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: app-dev-use-k8s-dashboard
subjects:
- kind: ServiceAccount
  name: app-dev
  namespace: dev-env
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view-addition-for-k8s-dashboard-access

Create the app-dev-use-k8s-dashboard clusterrolebinding

kubectl apply -f https://raw.githubusercontent.com/123MwanjeMike/k8s-rbac/main/clusterrolebindings/use-k8s-dashboard.yaml

Our app-dev service account can now be used by our "application developers" to create applications in the dev-env namespace and conveniently access the k8s dashboard.

Putting it to use

Create a kubectl configuration

We are going to need a token for the app-dev service account and so to get it, run the commands below

export NAMESPACE="dev-env"
export SERVICE_ACCOUNT="app-dev"
kubectl -n ${NAMESPACE} describe secret $(kubectl -n ${NAMESPACE} get secret | (grep ${SERVICE_ACCOUNT} || echo "$_") | awk '{print $1}') | grep token: | awk '{print $2}'\n

Your output will be some long string. It should start with something similar to what I have below. Save that token somewhere temporarily.

Next, get the certificate data by running the command below in the same terminal where you had defined the NAMESPACE and SERVICE_ACCOUNT variables.

kubectl  -n ${NAMESPACE} get secret `kubectl -n ${NAMESPACE} get secret | (grep ${SERVICE_ACCOUNT} || echo "$_") | awk '{print $1}'` -o "jsonpath={.data['ca\.crt']}"

Your output will be another long string whose beginning is something similar to what I have below. Save that certificate data somewhere temporarily.

With these two, we let's create our kubectl config file. You might already have a *.kube/config *which you might still need. So, we shall back it up as .kube/config.bak and create a new one for the app-dev. Run the commands below to do precisely that.

cd
mv .kube/config .kube/config.bak
nano .kube/config

This will open the nano text editor. Fill in the template below paste in the editor. Make sure to substitute the certificate-data, server-ip-address, cluster-name, and token with the correct values before saving the changes. Remember to delete the token and certificate data from where you had temporarily saved them.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data:  <certificate-data>
    server: https://<server-ip-address>:6443
  name: <cluster-name>
contexts:
- context:
    cluster: <cluster-name>
    namespace: dev-env
    user: app-dev
  name: <cluster-name>
current-context: dev-env
kind: Config
preferences: {}
users:
- name: app-dev
  user:
    client-key-data:  <certificate-data>
    token:  <token>

Access the Kubernetes dashboard

Run kubectl proxy and click here to access the dashboard.
Sign in with the .kube/config we created in the previous step.
Attempt to create a user role in the dev-env namespace with the manifest below. All should go on well.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-admin
  namespace: dev-env
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

Attempt to create the user role in the kube-public namespace with the manifest below.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-admin
  namespace: kube-public
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

Tadaa!

We did it. app-dev has no such rights outside the dev-env namespace. All application developers can now use that .kube/config file to gain controlled access to the cluster

Conclusion

In this post, we looked at how we can implement RBAC not only at the namespace level, but also at the cluster level. We specifically used a service account given that it is managed by Kubernetes taking away this task from us. It is also important to note that Kubernetes also user accounts as much as it does service accounts. One key difference between the two is that user accounts represent actual Kubernetes users unlike the service accounts.

I hope you enjoyed this article and that you'll show support of some sort as I'd greatly appreciate that. And if you have any thoughts or questions in relation to this, feel free to drop them via the comments section. I'd love to hear from you.

Happy hacking

Additional resources

What is best platform for tech blogging and building a developer community?

Mike Mwanje — Thu, 21 Jul 2022 21:58:00 +0000

As a technology professional, I have always been passionate about sharing knowledge with others. With the rise of blogs and social media, I realized that there are many other benefits to blogging besides just sharing knowledge. For example, by blogging, I am able to build visibility and grow an audience. Additionally, documenting my work and experiences can be a valuable resource for myself in the future.

Given that I am committed to blogging monthly, it was important for me to find the best platform for this. I wanted to take advantage of everything that blogging has to offer and this platform had to be feature rich and user-friendly. So I decided to carry out a research on different platforms that could assist me in achieving my goals. The platforms that I evaluated were Medium, DEV(Dev.to), and tech Twitter. You might be wondering about Hashnode but I will get to this in a bit.

If you were to choose one platform to blog on, which one would it be and why?

I picked thoughts from people both on Twitter and a couple of developer communities, and here is what I found...

DEV

DEV is a great place to start out as a technical blogger. It has a large technical audience and a supportive developer community. Articles from junior and senior developers get the same level of visibility. This makes it an ideal place for enthusiastic learners as well as seasoned professionals to share their learnings and expertise with a broader audience. DEV Markdown also makes it easy to write code snippets and have them formatted correctly.

Medium

Medium provides a greater visibility for content than DEV with a much more diverse audience. While it is not as popular with developers, Medium has a growing number of tech bloggers who use the platform to share their work and insights with a wider audience. Medium is however best to use if you already have an established blog audience and want to amplify your content as new bloggers may not get as much visibility. Medium is a better place for sharing experiences and ideas rather than just technical content. It is also a great place to create a thought leadership presence for yourself and build a brand.

Hashnode

Hashnode is relatively new to the market compared to other blogging platforms. However, it has risen quickly to become popular with developers because it offers all the features that tech bloggers require. I had initially not considered Hashnode as it is somewhat similar to DEV. Both are tailored specifically for the developer community and also have a similar markdown support (although Hashnode’s offerings are far superior to those available on DEV). Hashnode however has smaller audiences than DEV and Medium and as a result, you may not get as much attention.

Tech Twitter

Tech Twitter is where all the developers hang out. If you have not heard of it before, then you must be living under a rock! Tech Twitter is a micro blogging platform where developers share their latest discoveries, thoughts and insights on various tech topics. It is a noisy place where you need to be constantly talking to get noticed. However, it is a great place to build relationships with other developers and get your content read by the right people.

Conclusion

Choosing the right blog platform is not easy. It depends on your objective and how involved you want to be with the platform. However, if you are starting out and are looking to build an organic audience, then I would suggest DEV. If you are looking for a platform that provides greater visibility and a wider audience as a thought leader, then Medium the best choice for you. While if you want to take charge of your blogging process and build a more niche audience, then Hashnode would be the best platform for you. Finally, if you are looking for a platform to constantly keep your audience engaged, then Tech Twitter is the place to be.
The good news is that you can always use canonical urls to republish your blogs to leverage what the other platforms have to offer.

Do you have any thoughts on what I just shared? Let me know in the comments below! I would also love to hear from you on what your favourite blogging platform and why?

Happy blogging!

How to run API automated tests with Newman on CircleCI.

Mike Mwanje — Sat, 15 Jan 2022 17:10:48 +0000

Postman is a great tool not only for building, but also for testing APIs. In this post, we shall look at how to use Newman, Postman’s command-line collection runner, to run automated tests for an API in a CI/CD pipeline running on CicleCI. Whereas there maybe various tools and libraries to test APIs built in a specific language, automation testing with Newman cuts across and thus the steps followed herein can be followed to test an API developed in any programming language. That stated, we shall use a Node.js API in this post.

Tutorial

Setting up
Importing collection and environment
Integrating with CircleCI

Setting up

Before we start, make sure you have the newman cli installed on your computer. If you do not have it yet, install it globally by simply running

$ npm install -g newman

Fork and clone this repository which has the API we shall be using and change directory into the created folder. Run the commands below to do so.to install it globally.

$ git clone https://github.com/123MwanjeMike/node-express-realworld-example-app.git
$ cd node-express-realworld-example-app/

Switch to the 00--tutorialStart branch.

$ git checkout 00--tutorialStart

Install the dependencies with a package manager(npm, yarn) of your choice. In my case, I’ll be using yarn all throughout.

$ yarn install

Run the start script to confirm that everything is working fine.
Ps: You need to have MongoDB installed on your machine. If you don’t have it, head over here to install it or jump right to Importing collection and environment.

$ yarn start

Open another terminal and run the test script

$ yarn test

Importing collection and environment

Now let’s head over to Postman to get our hands dirty. We need to have our collection in Postman and its environment as well. So, open your postman Workspace and click the Import button. You can either import from the test folder of your cloned repository or from your forked repository(00--tutorialStart branch) on GitHub.

With the import complete, set the environment to Conduit API Tests — Environment. We can add one more test of our own. So head over to Auth/Login and Remember Token in the Conduit API Tests collection, select the Tests tab, and add a test to check that the response status code is 200 from the SNIPPETS on the right. Send the request.

Integrating with CircleCI

We shall now integrate our API in Postman with CircleCI. This way, we shall also be able to trigger and view test builds from within Postman. So, head over to APIs and click ‘+’ to create a new API.

Head over to the Test tab and click Add Test Suite. Select Add existing test, then select Conduit API Tests

Still at the same page, under Connect to CI/CD Builds, select CircleCI. You’ll be presented with a screen such as below.

We need to generate an API key for this integration on CircleCI. So click here to go to your CircleCI User Settings page, Personal Access Tokens section from where you’ll create one. Copy the API token you have added and paste it in the API Key input on the Postman screen above.

Enter a Nickname and select the CI Project to run the tests. It’s very likely your forked project is not on that list yet. You want to go back to your CircleCI dashboard and Set Up Project for it to reflect on Postman. It’s okay for the first build to fail.

Next we shall generate a Newman configuration that we shall place in our .circleci/config.yml file in the project repository. Click Configure Newman in the top right corner.

For our environment, we need to create a mock server so that we don’t run our requests against any real data. So click Mock Servers on the Postman left menu and click ‘+’ to create one. Choose the Select an existing collection option.

Select the Conduit API Tests collection. You’ll proceed to the Configuration screen as below.

Click the Create Mock Server button and copy the Mock URL provided.
Go to Environments to update the apiUrl variable for the Conduit API Tests — Environment. Set its INITIAL VALUE to the Mock URL you copied and save the changes.

With everything now set, we can head back to generate our Newman configuration. Set the collection to Conduit APT Tests and environment to Conduit APT Tests -Environment

Copy the generated configuration. We shall add it to .circleci/config.yml in our code base. So create the file in the project directory with the commands below.

$ mkdir .circleci
$ touch .circleci/config.yml

Paste the copied configuration in the created file. Create a workflow at the bottom to run your job. The workflow should be similar to that below.

workflows:
  test-workflow:
    jobs:
      - newman-collection-run

Your entire .circleci/config.yml file should look like that below. Take special care of the indention newman/newman-run step(lines 13 –15) to avoid indention related errors.

We are almost done. However, as you might have noticed, we have two instances of $POSTMAN_API_KEY (lines 14 and 15) in our .circleci/config.yml file and these must be substituted with an actual value on CircleCI. To create a Postman API Key, head over here and click Generate API Key. Copy the key and go to the project settings on CircleCI. Add an environment variable called POSTMAN_API_KEY and paste the Postman API Key in the value box.

While still on CircleCI, go to the Organization Settings > Security > Orb Security Settings, and select Yes to allow use of third-party orbs. This is because in our CircleCI configuration, we use newman: postman/newman@0.0.2 on line 5 which is not a default CircleCI orb.
Commit and push the changes to GitHub

$ git add .circleci/config.yml
$ git commit -m "add circleci config"
$ git push origin

Yeeey!
All our tests passed. We can also view our build history on Postman.

Conclusion

As you too have seen, testing your API with Postman is quite basic and you do not need to install any other testing packages into your project like mocha, jest, chai-http for automated testing. All you need is some basic understanding of JavaScript to write test cases on Postman.

I hope you enjoyed this post and wouldn't hesitate to leave me a heart or comment.

Happy coding!

AutoIdle cuts your Heroku bill by auto-putting your staging and review apps to sleep

Mike Mwanje — Wed, 27 Oct 2021 21:56:22 +0000

From simple to complex applications, Heroku stands out as a deployment choice for many developers. This is because with Heroku, getting an application up and running is a very simple procedure that abstracts the underlying infrastructure and its scaling needs. However, with more applications running on Heroku is a growing bill even when no traffic is being served. AutoIdle is a Heroku add-on that helps cut your Heroku bill by automatically putting your staging and review apps to sleep when you don't need them. In this article, we shall see how we can install AutoIdle on a Heroku app and review apps in a pipeline and observe how much we save.

How does AutoIdle work?

AutoIdle typically puts your apps to sleep after 30mins of inactivity and gets them up and running in under 10seconds of a new request. This way, you are not charged for the time the your apps are up but idle. With that stated…

…let's get our hands dirty.

Let's create an app called autoidle-saving in a Heroku pipeline. Starting from the start branch of this repository, we shall incrementally build our application to the state in the main branch; so fork the repository and go here to create a Heroku pipeline. You will be presented with the screen below.

Now select the Settings tab and under the Connect to GitHub option, search for the forked repository and connect it to the pipeline.

After connecting the repository, the Review apps option should now be available. Click Enable to have review apps for Pull Requests.
Note: Check the Wait for CI to pass option if you want HerokuCI to run your tests before deploying to the review app. There is a detailed article on Heroku CI and how to use it here that you can check out if you want to learn more about HerokuCI specifically.

Return to the pipeline tab now and for each of staging and production sections, click Add app, and click Create new app… to create a staging and production app respectively.

Your pipeline now looks like so

Let us make one additional tweak on our two apps. We want the staging app to automatically deploy code in the develop branch and the production app that in main branch on every push to the respective branches of the connected GitHub repository. So, starting with the staging app, click the Configure automatic deploys… option.

You will be presented with a screen such as below. Select the develop branch and click Enable Automatic Deploys

The same goes for the production application, only that for it we shall deploy the main branch.

The pipeline is now ready and we can make an addition to our app so as to set everything in motion.
Run the commands below to clone the forked repository and install the dependencies of the application.

$ git clone https://github.com/123MwanjeMike/autoidle-saving.git
$ cd autoidle-saving
$ npm install

Now to confirm that all is okay, run npm start in the terminal and you should have the output below

Now let us deploy to our staging app. So back to the staging section of the pipeline, and under the app options, click Deploy a branch…

…and deploy the develop branch.

Let us try to access our staging app in the browser.

We are now going to install AutoIdle on the staging app so that it is automatically put to sleep. So if you already downloaded and installed Heroku CLI on your computer, go to your terminal and login.

$ heroku login

Next, we shall add our staging app's git repository to the local repository. To get a Heroku app's git URL, select the app and go to its Settings tab. You should see the Heroku git URL listed under the app's information.

Copy the link and add it as a remote repository

$ git remote add staging-app https://git.heroku.com/autoidle-saving-staging.git
$ git remote -v

You should see a list of your remote repositories in the terminal.

Now we shall add AutoIdle to our application.

$ heroku addons:create autoidle

We shall also add AutoIdle to our review apps. So, create a new branch heroku-config and add an app.json file, a manifest format for describing Heroku web apps.

$ git checkout -b heroku-config

the app.json has the content below

{
   "environments": {
      "review": {
         "addons": ["autoidle:hobby"]
      }
   }
}

Now commit and push your changes to the GitHub repository.

$ git add app.json
$ git commit -m "AutoIdle configs for review apps"
$ git push -u origin heroku-config

Open a PR for the heroku-config branch against develop and merge it.

Now lets make a final addition to our app to have a review app and we compare the saving with AutoIdle. The endpoint is for the 'Hello World, happy saving!' message.

const express = require('express');

const app = express();

app.get('/', (req, res) => {
    res.status(200).json({ message: `Welcome!, let's save with AutoIdle 🎊`});
});

app.get('/salutation/:name', (req, res) => {
    res.status(200).json({ message: `Happy saving ${req.params.name}.`});
});

app.get('/world', (req, res) => {
    res.status(200).json({ message: 'Hello World, happy saving!'});
});

app.get('*', (req, res) => {
  res.status(404).json({ message: 'Sorry, resource not found.' });
});

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`Listening on port ${port}`));

Commit and push your changes.

$ git commit -am "hello-world"
$ git push -u origin hello-world

Open a PR for hello-world against develop and we should have a new review app for that.

Now let's observe the saving on the AutoIdle dashboard. In your terminal, run

$ heroku addons:open autoidle

You will be taken to the browser and presented with the screen below.

We can see that all our apps are running and we don't have any available saving data, so shall check again after some time has passed.

We can see that both our staging and review applications were automatically stopped without us having to click any button.

Conclusion

With cloud computing, every extra second your application is running counts and you want to be as lean as possible. In this tutorial, we only used AutoIdle on two applications, but imagine how much you could save with a large application with multiple contributors and new PRs created by the minute, each with a new review app. The cost of having apps running while not actively in use can be unnecessarily large and overwhelming. I hope AutoIdle is a tool you and your organization can leverage to reduce costs.

Till next time,
Happy saving!

Build a CI/CD pipeline with Heroku CI.

Mike Mwanje — Fri, 01 Oct 2021 20:56:22 +0000

Heroku CI automatically runs your app’s test suite with every push to your app’s GitHub repository, enabling you to easily review test results before merging or deploying changes to your codebase.

Besides strong Dev/prod parity, using Heroku CI comes with many benefits key of which include;

Parallel Test Runs.
Browser tests and User Acceptance Tests
Seamless integration with Heroku pipelines
Easy customization of test dependencies in the test environment
A Simple Integrated Solution (integration, hosting and deployment)
Simple, prescriptive developer experience in modern CI/CD solutions
Management support
Many supported languages

In this article, we shall build a Heroku Flow CI/CD pipeline that utilizes Heroku CI.

Part 1: App init.

We are going to build a simple node.js application that calculates the factorial of a number. We shall however start with its api and then add the factorial functionality later. This is so we can see Heroku CI in action.
To start, fork the repository here and then clone your forked repository to your computer.

$ git clone https://github.com/123MwanjeMike/cicd-with-herokuci.git

Next, change to the created directory and switch to the start branch.

$ cd cicd-with-herokuci/
$ git checkout start

Initialize the project with npm init and then install express

$ npm init -y
$ npm install express

You should have a new file package.json in your directory. Open it and add the start script node index.js.

"scripts": {
  "test": "echo \"Error: no test specified\" && exit 1",
  "start": "node index.js"
},

Now create a simple api for the app in index.js file with the code below.

$ touch index.js

const express = require('express');

const app = express();

app.get('/', (req, res) => {
  res.status(200).json({ message: 'Welcome to the Factorial calculator 🎊' });
});

app.get('*', (req, res) => {
  res.status(404).json({ message: 'Resource not found.' });
});

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`Listening on port ${port}`));

Running npm start should give a similar output as below

We now have a basic setup for our application.

Part 2: The pipeline.

We shall be adding our application to a Heroku pipeline.
First we need to tell Heroku which services to run and how to run these. To do this, we shall add a Procfile file at the root of our directory by running;

$ echo "web: node index.js" >> Procfile

Commit and push your changes to the remote repository.

$ git add .
$ git commit -m "<enter your message here>"
$ git push origin start

Now move here if you have a Heroku account and create a new Heroku pipeline. Remember to connect your forked repository as you create the pipeline. You should end up with a screen like below.

Click Enable Review Apps to have deploy previews for the pull requests(PRs) before they are merged.

Next, under the staging section of the pipeline, click Add app and create a new app. Its common practice to have the staging app deployed from the develop branch and the production app from the main branch.
In this case, my factorial-app is being deployed from my default branch(main).

Click settings and scroll to the Heroku CI section. Click Enable Heroku CI.
Note: Heroku gives 1000 free dyno hours per month to its verified users and these can be used to access the Heroku CI service for free until they are depleted — in which case you start paying per extra dyno minutes used.

Our pipeline is now set up and we are ready for action.

Part 3: The application’s factorials service

We are now going to add a service to our application. We want to enable it calculate the factorial of a number. This will be through a get request with path parameters. We shall use the test driven development approach while at it, so install the test runner mocha and the assertion library chai.

$ npm install -D mocha chai

Open the package.json file and find the test key under scripts and change its value to have mocha *.test.js || true. Your scripts now look like so.

"scripts": {
  "test": "mocha *.test.js || true",
  "start": "node index.js"
},

We are basically telling mocha to execute .test.js files without displaying errors when we run npm test.
Let us now create our test file factorial.test.js that will have our tests which will be automated later on.

$ touch factorial.test.js

const { assert } = require('chai');
const factorial = require('./factorial');

describe('Factorial test', () => {
  it('Factorial(0) = 1', () => {
    assert.equal(factorial(0), 1);
  });

  it('Factorial(1) = 1', () => {
    assert.equal(factorial(1), 1);
  });

  it('Factorial(5) = 120', () => {
    assert.equal(factorial(5), 120);
  });

  it('Factorial(171) = Infinity', () => {
    assert.equal(factorial(171), Infinity);
  });
});

Next, create the file factorial.js to which we shall write our function that calculates the factorial of a number.

$ touch factorial.js

const factorial = (number) => {
  let result = 1;
  if (number === 0 || number === 1) {
    return result;
  }
  for (let i = number; i >= 1; i -= 1) {
    result *= i;
  }
  return result;
};

module.exports = factorial;

We can run our test suite with npm test and get the output as below

Now let us update our index.js too with an end point that can serve requests for the factorials of numbers.

const express = require('express');
const factorial = require('./factorial');

const app = express();

app.get('/', (req, res) => {
  const { host } = req.headers;
  res.status(200).json({ message: 'Welcome to the Factorial calculator 🎊', docs: `http://${host}/docs` });
});

app.get('/docs', (req, res) => {
  const { host } = req.headers;
  res.status(200).json({
    message: 'Documentation',
    request: `http://${host}/factorial/<number>`,
    response: 'The factorial of <number> is <result>`',
    example: {
      request: `http://${host}/factorial/5`,
      response: 'The factorial of 5 is 120`',
    },
  });
});

app.get('/factorial/:number', (req, res) => {
  const { number } = req.params;
  if (isNaN(number)) return res.status(400).json({ message: `'${req.params.number}' is not a number.` });
  if (number > 200) return res.status(200).json({ message: `The factorial of ${number} is Infinity` });
  return res.status(200).json({ message: `The factorial of ${number} is ${factorial(number)}` });
});

app.get('*', (req, res) => {
  const { host } = req.headers;
  res.status(404).json({ message: 'Resource not found.', docs: `http://${host}/docs` });
});

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`Listening on port ${port}`));

I added a docs endpoint to provide documentation for our service request.

Part 4: The magic

The moment of truth has come. Now we shall push our changes again to GitHub, open a pull request then observe how it all works.

$ git add .
$ git commit -m "Factorial service"
$ git push origin start

After opening the pull request, we see that Heroku CI tests passed and that the branch was successfully deployed. We can also go ahead to checkout the deployment.

Finally, lets enable automatic deploys for our factorial-app still under the staging section of our pipeline.

From now on, any code push to our main branch will automatically be deployed to the staging environment of our application.

Since everything is working fine, we can then move/promote the app to production.

Conclusion

As you have seen, setting up a CI/CD pipeline with Heroku CI is quite easy and Heroku Flow is a very intuitive workflow for visualizing your code delivery process. Heroku is ideal for small and medium business applications even though large business application with a microservices architecture can still leverage it. With Heroku CI, you can move fast without breaking things.

For reference, you can find the entire project code in this repository with branches start, part-1, part-2, and part-3 that have the resultant code version of each respective part above and the final application code in the main branch.

This post was sponsored by AutoIdle. AutoIdle is an add-on that cuts your Heroku bill by automatically putting your staging and review apps to sleep when you don't need them.

Happy hacking.

Project management with Github Issues.

Mike Mwanje — Mon, 06 Sep 2021 15:23:41 +0000

Software projects might be different from other types of projects in terms of visibility, complexity, conformity, and flexibility as Fred Brooks points out; but one thing in common with all projects is change. Essentially, project management is change management and this is what project management tools help us with. Tools like Trello, ClickUp, Jira are built to enable us best plan and track our work - herein, you'll learn to do the same with Github Issues.

But first, why Github Issues?

Given that Github is a leading software development platform, most project management tools support integration with it as a way to bridge management and development teams. However, the underlying problem still remains: this is not where the code is and developers have to leave the platform to update their work item status on these project management tools. In addition, automation on these is not as easy and the pricing of these tools is maybe high.

What are Github Issues.

Github Issues is a collection of Github products that include labels and milestones, issues, project boards, and projects(beta). All these products can be used together to easily plan and track your progress on your software project. The scope of this article will be limited to issues and Github project boards and that's what will be covered in the tutorial section.

Github project boards

Project boards on GitHub help you organize and prioritize your work. You can create project boards for specific feature work, comprehensive roadmaps, or even release checklists. With project boards, you have the flexibility to create customized workflows that suit your needs.

Depending on your needs, you can either create repository, user-owned, or organization-wide project boards all with different scope. A card on a project board represents either an issue, a pull request or a note. These cards contain metadata about the issue or pull request the like labels, assignee(s), the status, and who opened it. Github project boards also provides you with templates to create a new board some of which come with built in automation. They include basic kanban, automated kanban, automated kanban with review, and bug triage. Now, let's see how this works.

Tutorial

Step 1: Create the project board.
First, create a repository project board from the Automated Kanban template under the project template drop-down. I'll call my board User App.

(Optional) Rename the To do Column to Product Backlog and add a column called Sprint Backlog with the To do preset, and click the Reopened checkbox under Move issues here when… and move it after the Product Backlog column.

Step 2: Add notes to the project board
Next, we will add notes to our project board under the Product Backlog list and after convert them to issues. These are features we plan to work on.

Step 3: Assign issues.
Now that we have a good understanding of what we plan to do, we shall start embarking on the issues. So say, for the first sprint we are working on Signup with email and Login with email and password. We shall move these cards to the Sprint Backlog column and assign each issue to an individual.

Step 4: Update In Progress column automation.
We will now update automation for the In Progress column move pull requests here when newly added, reopened, and pending approval by reviewer. This will enable us know what tasks are currently being worked upon.

Moment of truth
With the above setup complete, we are now good to go. When an assignee opens a PR(Pull Request) and links it to the issue, we shall see a new card under the In Progress column that corresponds to the PR, and when the PR is merged, both the issue card and PR card will automatically be moved to the Done column.

Github project board: https://github.com/123MwanjeMike/user-app/projects/1

Final thoughts.

This is only the tip of the iceberg that can be done with Github Issues and with good mastery of the Github project management products, you are spoilt for choice.
One only challenge I’ve found with Github Issues is that clients and management may have little or totally no knowledge of how to use these products. In such cases, you want to rethink the solution to get a workaround.

How different is CommonJs require from ES6 import?

Mike Mwanje — Sat, 28 Aug 2021 09:28:36 +0000

In JavaScript, you can use either ECMAScript 6(ES6) modules or CommonJs modules in your project and there are a few differences between these that do affect how your program modules are loaded. In this article, I explore how each works and how it may affect your program execution.

CommonJs modules.

CommonJs is the original and default module system of Node.js which uses require and module.exports. Below is an example.

// Importing modules
const fs = require('fs');
const fileDelete = require('./fileDeleter');
const fileName = require('./fileNamer');

const writeFile = (data) => {
  return fs.writeFileSync(fileName, data);
}

// Exporting writeFile module
modules.exports = writeFile;

With require, you can’t selectively load only the modules you need. This means even the fileDelete module from the example above will be imported even if it is not needed or used anywhere. Additionally, importing of the modules is synchronous which means that fileName module can’t be imported before fs and fileDelete modules are imported, and a failure to import fileDelete will cause run-time errors even if it is not used anywhere in our program. CommonJS modules are the choice for the node.js server.

ECMAScript modules

ECMAScript modules are relatively newer and use import and export. Below is the transformation of our CommonJs example from above to ESM.

// Importing modules
import fs from 'fs';
import fileDelete from './fileDeleter';
import fileName from './fileNamer';

const writeFile = (data) => {
  return fs.writeFileSync(fileName, data);
}

// Exporting writeFile module
export default function writeFile;

With import, you load only the modules you need. For example, the fileDelete module from the above will not be imported since it is not used anywhere. Additionally, the importing of the modules is asynchronous which means that both fs and fileName are imported at the same time. You generally want to use ESM for your new projects.

…how about .cjs and .mjs?
.cjs is a file extension for CommonJS modules while .mjs is a file extension for ECMAScript module. Node.js by default treats .js files as CommonJS modules. You can change this by adding "type": "module"to your package.json file so you can use ECMAScript modules (in your .mjs files) within a Node.js environment. This is what Google Chrome V8 recommends.

I hope this was helpful to you and for further reading, do checkout JavaScript modules.

Happy coding!