DEV Community: MyCareersFuture

Setting up ZSH on Android

Kai Hong — Sun, 20 Dec 2020 09:50:59 +0000

One day I was out and about seizing the day, when I suddenly saw on the news that a critical zero-day bug has been unleashed and I urgently needed to patch my servers, but I don't have my laptop with me!

Has that ever happened to you? Well me neither, but in case you ever did, here's how you can setup a proper ZSH terminal on your Android device!

What you need

1 x Android device
1 x Internet Connection
2 x 1 min

Steps 🚶

# Step 1
pkg install zsh

# Step 2 (https://ohmyz.sh/#install)
sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Yeap that's all you need to get ZSH on your android device. For any other packages, you can use pkg or apt-get. Termux provides a slew of useful utilities by default, and you can expand upon that with Termux APIs; allowing you to do things like retrieving SMS, getting location, etc. You could use SMS as an out-of-band way of triggering a smart home action for example.

Note that in order to use Termux APIs, you first need to install from the Play Store, then run pkg install termux-api. Then make sure that the Termux API application has enough permissions to do what you want.

But wait, you don't like the limited Linux functionality that Termux provides you? How about installing Ubuntu on your phone?. Someone created PRoot, which is a user-space implementation of chroot, which is what makes running Ubuntu on Android pretty easy and straightfoward (unlike the janky dual-boot script days).

My setup 🤓

Having plain old vanilla ZSH is fine and all, but you can spice your life up with some useful plugins. To make installation easier, I rely on a plugin manager called Antibody. To get started, follow the instructions on their site cause it's really well written.

Using the static loading method (because it's faster), this is my list of plugins in my .zsh_plugins.txt file.

zsh-users/zsh-completions
zsh-users/zsh-syntax-highlighting
zsh-users/zsh-history-substring-search
zsh-users/zsh-autosuggestions
mafredri/zsh-async
sindresorhus/pure

I think the plugins are pretty self-explanatory except for the last two, which is the theme that I use because it's minimalistic.

Two other plugins I use are

autojump: quickly switch between directories based on history (had to manually install)
scmpuff: numbered git files and nice alises. I had to compile this, which forced me to install golang (it works!)

My use case 🦄

Why go through all this effort to have a nice shell experience on Android? All of these was done to set me up for having a Productive 2 weeks in reservist. I wanted to try if it is possible to develop on an Android Tablet.

Spoiler: it worked bloody well

Article on developing on a tablet will be coming soon where I also share how it was also useful for making quick changes on my portfolio site, as well as miscellaneous SSH tasks on my VPS.

In summary, it's really easy to setup a decent shell experience on your Android devices these days (hint: iOS too 😏). Have fun!

Getting started with grafana and prometheus for kubernetes metrics

Ryan — Thu, 03 Sep 2020 07:56:59 +0000

In the previous post, we've gotten Grafana up and running with a cloudwatch datasource. While it provides us with many insights on AWS resources, it doesn't tell us how our applications are doing in our Kubernetes cluster. Knowing the resources our applications consume can help prevent disasters, such as when applications consume all the RAM on the node, causing it to no longer function, and we now have dead nodes and applications.

For us to view the metrics on our Grafana dashboard, we can integrate it into a Prometheus datasource, and have Prometheus collect metrics from our nodes and applications. We will deploy Prometheus using helm, and explain more along the way.

Requirements
The quick 5-minute install
- What you get
- Setup
- Storage space
- Installation
- Connecting Grafana to Prometheus
- Adding Dashboards
How does it work?
- Node Metrics
- Application Metrics
- So how is this configured?
Wrapping up

Requirements

Kubernetes cluster, preferably AWS EKS
Helm

The quick 5-minute install

What you get

Prometheus Server
Prometheus Node Exporter
Prometheus Alert Manager
Prometheus Push Gateway
Kube State Metrics

Setup

Storage space

Before we begin, it is worth mentioning the file storage requirements of Prometheus. Prometheus server will be running with a persistent volume(PV) attached, and this volume will be used by the time-series database to store the various metrics it collects in the /data folder. Note that we have set our PV to 100Gi in the following line of values.yaml.

https://github.com/ryanoolala/recipes/blob/8a732de67f309a58a45dec2d29218dfb01383f9b/metrics/prometheus/5min/k8s/values.yaml#L765
## Prometheus server data Persistent Volume size
    ##
    size: 100Gi

This will create a 100Gib EBS volume attached to our prometheus-server. So how big of a disk do we need to provide? There are a few factors involved so generally, it will be difficult to calculate the right size for the current cluster without first knowing how many applications are hosted, and also to account for growth in the number of nodes/applications.

Prometheus also has a default data retention period of 15 days, this is to prevent the amount of data from growing indefinitely and can help us keep the data size in check, as it will delete metrics data older than 15 days.

In Prometheus docs, they suggest calculating using this formula, with 1-2 bytes_per_sample

# https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects
needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

This is difficult for me to calculate even with an existing setup, so if this is your first time setting up I can imagine it being even more so. So as a guide, I'll share with you my current setup and disk usage, so you can gauge how much of disk space you want to provision.

In my cluster, I am running

20 EC2 nodes
~700 pods
Default scrape intervals
15 day retention

My current disk usage is ~70G.

If the price of 100GiB of storage is acceptable for you, in my region it is about USD12/month. I think it is a good starting point and you can save the time and effort on calculating for storage provisioning and just start with this.

Note that I'm running Prometheus 2.x, that has an improved storage layer over Prometheus 1, and has shown to have reduced the storage usage and thus the lower need of disk space, see blog

With this out of the way, let us get our Prometheus application started.

Installation

$ helm install prometheus stable/prometheus -f https://github.com/ryanoolala/recipes/blob/master/metrics/prometheus/5min/k8s/values.yaml --create-namespace --namespace prometheus

Verify that all the prometheus pods are running

kubectl get pod -n prometheus
NAME                                             READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-78b5c64fd5-ch7hb         2/2     Running   0          67m
prometheus-kube-state-metrics-685dccc6d8-h88dv   1/1     Running   0          67m
prometheus-node-exporter-8xw2r                   1/1     Running   0          67m
prometheus-node-exporter-l5pck                   1/1     Running   0          67m
prometheus-pushgateway-567987c9fd-5mbdn          1/1     Running   0          67m
prometheus-server-7cd7d486cb-c24lm               2/2     Running   0          67m

Connecting Grafana to Prometheus

To access Grafana UI, run kubectl port-forward svc/grafana -n grafana 8080:80, go to http://localhost:8080 and log in with the admin user, if you need the credentials, see the previous post on instructions.

Go to the datasource section under the settings wheel

and click "Add data source"

If you've followed my steps, your Prometheus setup will create a service named prometheus-server in the prometheus namespace. Since Grafana and Prometheus are hosted in the same cluster, we can simply use the assigned internal A record to let Grafana discover prometheus.

Under the URL textbox, enter http://prometheus-server.prometheus.svc.cluster.local:80. This is the DNS A record of prometheus that will be resolvable for any pod in the cluster, including our Grafana pod. Your setting should look like this.

Click "Save & Test" and Grafana will tell you that the data source is working.

Adding Dashboards

Now that Prometheus is setup, and has started to collect metrics, we can start visualizing the data. Here are a few dashboards to get you started.

https://grafana.com/grafana/dashboards/315
https://grafana.com/grafana/dashboards/1860
https://grafana.com/grafana/dashboards/11530

Mouse over the "+" icon and select "Import", paste the dashboard ID into the textbox and click "Load"

Select our Prometheus datasource we added in the previous step into the drop-down selection

You will have a dashboard that looks like this

You may have noticed the "N/A" in a few of the dashboard panels, this is a common problem in various dashboards, due to incompatible versions of Prometheus/Kubernetes with changes in metric labels, etc.
We will have to edit the panel and debug the queries to fix them. If there are too many errors, I will suggest finding another dashboard until you find one that works and fits your needs.

How does it work?

You may have wondered how all these metrics are available, even though you've simply deployed it, without configuring anything other than disk space. To understand the architecture of Prometheus, check out their documentation. I've attached an architecture diagram from the docs here for reference.

We will keep it simple by only focusing on how we are retrieving node and application(running pods) metrics. Remember at the start of the article, we noted down the various Prometheus applications you will get for following this guide.

Prometheus-server pod will be pulling metrics via Http endpoints, most typically the /metrics endpoint from various sources.

Node Metrics

When we installed prometheus, there is a prometheus-node-exporter daemonset that is created. This ensures that every node in the cluster will have one pod of node-exporter, which is responsible for retrieving node metrics and exposing them to its /metrics endpoint.

Application Metrics

prometheus-server will discover services through the Kubernetes API, to find pods with specific annotations. As part of the configuration of the application deployments, you will usually see the following annotations in various other applications.

metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "4000"
    prometheus.io/scrape: "true

These are what prometheus-server will look out for, to scrape for metrics from the pods.

So how is this configured?

Prometheus will load its scraping configuration from a file called prometheus.yml which is a configmap mounted into prometheus-server pod. During our installation using the helm chart, this file is configurable inside the values.yaml, see the source code at values.yaml#L1167. The scrape targets are configured in various jobs and you will see several jobs configured by default, each catering to a specific configuration of how and when to scrape.

An example for our application metrics is found at

#https://github.com/ryanoolala/recipes/blob/8a732de67f309a58a45dec2d29218dfb01383f9b/metrics/prometheus/5min/k8s/values.yaml#L1444
 - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
          - role: pod

        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name

This is what configures prometheus-server to scrape pods with the annotations we talked about earlier.

Wrapping up

With this, you will have a functioning metric collector and dashboards to help kick start your observability journey in metrics. The Prometheus we've set up in this guide will be able to provide a healthy set up in most systems.

However, there is one limitation again to take note of and that is this Prometheus is not set up for High Availability(HA)

As this uses an Elastic Block Store(EBS) volume, as we have explained in the previous post, it will not allow us to scale-out the prometheus-server to provided better service uptime, if the prometheus pod restarts, possibly to due Out-of-memory(OOMKilled) or unhealthy nodes, and if you have alerts set up using metrics, this can be an annoying problem as you will lose your metrics for the time being, and blind to the current situation.

The solution to this problem is something I have yet to deploy myself, and when I do, I will write the part 3 of this series, but if you are interested in having a go at it, check out thanos.

Hope that this has been simple and easy enough to follow and if you have a Kubernetes cluster, even if it is not on AWS, this Prometheus setup is still relevant and be deployed in any system, with a StorageDriver configured to automatically create persistent volumes in your infrastructure.

Getting started with deploying grafana and cloudwatch metric dashboards

Ryan — Wed, 02 Sep 2020 06:10:46 +0000

The Pillar of Metrics

Metrics is one of the key components in observability which is increasingly more important as we adopt more distributed application architectures, monitoring the health of our applications becomes difficult to manage if we don't have an aggregation system in place. If you are just starting on your observability journey, and find justifying for paid SaAS services such as datadog or splunk a tough barrier to overcome, you can easily start by first using open source solutions that can give you a better grasp of how metric collection works, and create dashboards to provide some insights into your current system.

In this post, we will be going through some quick recipes that help deploy Grafana onto an AWS Elastic Kubernetes Service(EKS) cluster with minimal effort, with dashboards created by the community. So let us get started.

Requirements
The quick 5-minute build
- What you get
- Setup
- Cloudwatch IAM Role
- Grafana
  - Installing Grafana
- Show me the UI!
- Whats next?
The 10-minute build
- What you get
- Setup
- Postgres RDS
- Grafana
Logging in
Wrapping up

Requirements

Kubernetes cluster, preferably AWS EKS
Helm
Terraform

The quick 5-minute build

What you get

Grafana instance
Cloudwatch metrics
Cloudwatch dashboards to monitor AWS services(EBS, EC2, etc.)

Setup

ryanoolala / recipes

A collection of recipes for setting up observability toolings

recipes

A collection of recipes for setting up resources in AWS and EKS

This is my attempt of trying to introduce observability tools to people and providing a recipe for them to add them into their infrastructure as easily as possible, as such you may find that most of these setups may be too simple for your production needs(e.g HA consideration, maintenance processes), and if I am able to think of ways to make these better and able to simplify into recipes, I will update this repository, as a recipe guide for myself in my future setups.

Requirements

This repository assumes you already have the following tools installed and required IAM permissions(preferable an admin) to use with terraform

terraform >= v0.12.29
terragrunt >= v0.23.6
kubectl >= 1.18
helm >= 3.3.0

Note

This is not a free tier compatible setup and any costs incurred will be bared by you and you…

View on GitHub

Clone the recipe repository from github.com/ryanoolala/recipes, which I will be using to reference the setup throughout this post

Cloudwatch IAM Role

We first create an IAM role with permissions to get metrics from cloudwatch, and to speed things up we'll be using terraform to provision the role and in my case, I'll be making use of terragrunt, but you can easily copy the inputs and into a terraform module variable input instead.

module "cloudwatch-iam" {
  source = "git::https://gitlab.com/govtechsingapore/gdsace/terraform-modules/grafana-cloudwatch-iam?ref=1.0.0"
  allow_role_arn = arn:aws:iam::{{ACCOUNT_ID}}:role/ryan20200826021839068100000001
  name = "ryan"
}

The grafana cloudwatch iam module takes in a EKS ARN role, this is because we want our Grafana application running on the node, to be able to assume this cloudwatch role, and be authorized to pull metrics from AWS APIs. This provides a terraform output of
grafana_role_arn = arn:aws:iam::{{ACCOUNT_ID}}:role/grafana-cloudwatch-role-ryan

Grafana

Here is where it gets interesting, we will be deploying Grafana using helm 3. Make sure you have your kubectl context set to the cluster you want to host this service on, and that it also belongs to the same AWS account which we just created the IAM role.

We create a datasource.yaml file with the following values, be sure to replace assumeRoleArn with your output from above.

# file://datasource.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Cloudwatch
        type: cloudwatch
        isDefault: true
        jsonData:
          authType: arn
          assumeRoleArn: "arn:aws:iam::{{ACCOUNT_ID}}:role/grafana-cloudwatch-role-ryan"
          defaultRegion: "ap-southeast-1"
          customMetricsNamespaces: ""
    version: 1
    # <bool> allow users to edit datasources from the UI.
    editable: true

This will allow grafana to start with a cloudwatch datasource that is set to use assumeRoleArn for retrieving cloudwatch metrics.

Installing Grafana

$ helm install grafana stable/grafana -f https://github.com/ryanoolala/recipes/blob/master/metrics/grafana/5min/k8s/grafana/values.yaml -f datasource.yaml --create-namespace --namespace grafana

or if you have cloned the repository, place datasource.yaml into ./metrics/grafana/5min/k8s/grafana and run

$ cd ./metrics/grafana/5min/k8s/grafana && make install.datasource

In a few moments, you will have a grafana running

$ kubectl get pod -n grafana
NAME                          READY   STATUS     RESTARTS   AGE
grafana-5c58b66f46-9dt2h      2/2     Running    0          84s

and to get access to the dashboard, run

$ kubectl port-forward svc/grafana -n grafana 8080:80
Forwarding from 127.0.0.1:8080 -> 3000
Forwarding from [::1]:8080 -> 3000

Show me the UI!

Navigate to http://localhost:8080 and you will see your Grafana UI

If you are wondering where these dashboards are loaded, I found them on grafana's dashboard site, picked a few of them, and loaded them by configuring values.yaml

# https://github.com/ryanoolala/recipes/blob/master/metrics/grafana/5min/k8s/grafana/values.yaml#L364

dashboards:
  default:
    aws-ec2:
      url: https://grafana.com/api/dashboards/617/revisions/4/download
    aws-ebs:
      url: https://grafana.com/api/dashboards/11268/revisions/2/download
    aws-cloudwatch-logs:
      url: https://grafana.com/api/dashboards/11266/revisions/1/download
    aws-rds:
      url: https://grafana.com/api/dashboards/11264/revisions/2/download
    aws-api-gateway:
      url: https://grafana.com/api/dashboards/1516/revisions/10/download
    aws-route-53:
      url: https://grafana.com/api/dashboards/11154/revisions/4/download
    aws-ses:
      url: https://grafana.com/api/dashboards/1519/revisions/4/download
    aws-sqs:
      url: https://grafana.com/api/dashboards/584/revisions/5/download

Whats next?

There are some limitations to what we have just deployed, while the UI allows to edit and even add new dashboards, the changes we make are not persistent, since we did not provide any persistent store for this setup. Let's make it better!

The 10-minute build

To save our changes, there are several ways to do so, the easiest probably being attaching a block store(EBS) volume to the instance, and have settings stored on the disk. However as EBS is not a ReadWriteMany storage driver, we cannot scale-out our Grafana instance across availability zones and different EKS nodes. The next easiest solution, in my opinion, will be to make use of AWS Relational Database Service(RDS), which is fully managed, with automatic backups and High Availability(HA), as our persistence layer for Grafana.

What you get

HA Grafana with persistence

Setup

Postgres RDS

We will be using postgres in this example, although Grafana supports MySQL and sqlite3 as well. In order to not digress, I will omit the setup instructions for the database, if you will like to know how I used terraform to deploy the instance, you may read up more in my README.

If you are not familiar with terraform, this might get slightly complicated, thus I will suggest that you create the postgres using the AWS console which will be much easier and faster, to keep it under the 10min effort required for this.

Grafana

Now that we have a postgres database setup, we will create a kubernetes secret object to contain the credentials needed for connecting to it.

$ cd ./metrics/grafana/10min/k8s/grafana
$ make secret
Removing old grafana-db-connection...
secret "grafana-db-connection" deleted
Postgres Host?: 
mydbhost.com
Postgres Username?: 
myuser
Postgres Password? (keys will not show up in the terminal): 
Attempting to create secret 'grafana-db-connection'...
secret/grafana-db-connection created

This secret grafana-db-connection will be used in our values.yaml and we will also set the environment GF_DATABASE_TYPE to postgres.

# https://github.com/ryanoolala/recipes/blob/cf7839e9e919735c72fee77450d891f8ee13ef17/metrics/grafana/10min/k8s/grafana/values.yaml#L268
## Extra environment variables that will be pass onto deployment pods
env:
  GF_DATABASE_TYPE: "postgres"

# https://github.com/ryanoolala/recipes/blob/cf7839e9e919735c72fee77450d891f8ee13ef17/metrics/grafana/10min/k8s/grafana/values.yaml#L282
envFromSecret: "grafana-db-connection"

With these changes done, we can upgrade our current deployed Grafana using helm upgrade grafana stable/grafana -f values.yaml --namespace grafana, or if you are starting from a fresh setup,

$ helm install grafana stable/grafana -f https://github.com/ryanoolala/recipes/blob/master/metrics/grafana/10min/k8s/grafana/values.yaml --create-namespace --namespace grafana

or if you have cloned the repository, run

$ cd ./metrics/grafana/10min/k8s/grafana && make install

This new Grafana will allow you to make changes to the system, add datasources and dashboards, and save these changes in the database so you don't have to worry about your instance restarting and having to start all over again.

Logging in

To make edits, you have to login using the admin user, the default username and password can be set during installation, by modifying the following in values.yaml

# Administrator credentials when not using an existing secret (see below)
adminUser: admin
adminPassword: strongpassword

After Grafana has been started, we can change our password on the UI instead, and the new password will be stored in database for future login sessions.

Wrapping up

Hopefully, this gave you an idea of how you can make use of the grafana helm chart and configure it to display cloudwatch metric dashboards.

In the next part, I will share more about deploying prometheus, which will provide us with more insights within the kubernetes cluster, including CPU/RAM usage of the ec2 nodes, as well as pods. These pieces of information will help us better understand our deployed applications.

Gitlab MR Bot: Getting people to do code reviews

Kai Hong — Mon, 03 Aug 2020 05:09:07 +0000

Hello World, recently at MyCareersFuture Team, we have a growing stack of merge requests (MR) that are missing the code review attention it needs to go into main branch. This slows down our delivery process as our working agreement is that every merge requires at least two other developers to approve first.

It's a long weekend, let's do something about it.

Goal: Get more eyes on the MRs ready for review.

MR reviews in MCF

Yes we really do review everything that goes in.

High quality code is important to us, and code reviews is one of the ways we uphold that standard. By ensuring at least two developers review each MR, we can catch potential bugs and inefficiency early in the cycle.

This also facilitates knowledge transfer between devs, which also translates to better code being written subsequently. All parties including the stakeholders understand the additional overhead to this process and have accepted it as we are aligned to the goal of delivering a quality product.

Why aren't more people reviewing?

Slack is our currently communication channel, and when a MR is ready for review, it is labelled for review, and posted to the channel for anyone to pick it up.

Sometimes all devs are just busy and missed the messages
Sometimes the request get buried among other conversations
Other factors like unfamiliarity with various codebases

Temporary solution

For the past 2 sprints or so, our kind scrum master has been helping to manually consolidate the various MRs and pinging the channel for people to take action on those items.

And it works! More devs have been noticeably more active in MR reviews ever since she started doing that.

So... let's automate that! 😉

Introducing the MR Bot

Objective: consolidate a list of opened merge requests that have a label of review me across multiple repositories daily and notify on slack.

We are using a self-hosted version of Gitlab, so we rely on labels for approval. For example, if I want to review a specific MR, I will add my label Review by Kai Hong, and subsequently Approved by Kai Hong.

The bot would only run once a day to prevent it from being spammy because, we wouldn't want people muting the channel do we? Since this runs daily, it sounds a lot more like a cronjob than a long running service. So let's model it as a batch process and build it!

A typical batch process chains a bunch of processors, where the output of one processor will be the input of the next processor.

Input -> Process -> Output

Let's take a look at the overview of the entire process before going into details. This "bot" is built with native NodeJS without any frameworks because it's only dealing with network requests.

const fetchTasks = GITLAB_PROJECT_ID_LIST.map(id => fetchMergeRequests(id));
Promise.all(fetchTasks)
  // Part 1
  // Input: list of fetch tasks
  // Process: resolve all requests
  // Output: list of json responses
  .then(res => {
    return Promise.all(res.map(r =>{
      return r.json();
    }))
  })
  // Part 2
  // Input: list of json responses
  // Process: extract/transform relevant data
  // Output: list of processed MR
  .then(listOfMrList => {
    // ...
    return mergeRequests.map(mr => processMr(mr));
  })
  // Part 3
  // Input: list of processed MR
  // Process: send to slack
  // Output: none
  .then(processedMrList => {
    sendToSlack(processedMrList);
  })
  .finally(() => {
    console.log(`stub end`);
  })
  .catch(err => console.error(`stub error`));

Part 1: Fetch data

The magic Gitlab API URL that allows me to fetch all the merge requests from a repo.
https://<gitlab_url>/api/v4/projects/<id>/merge_requests/state=opened&labels=Review+Me

In order to fetch from a list of repos, I had to brush up my async/await/promises codefu to make this work as it's not everyday that I try to synchronise a bunch of asynchronous calls to work with a batch process. And it was all solved with Promise.all([]).

It helps to consolidate all of the promises into one promise, and you just have to handle the output of that one, which made the batch process a lot simpler.

Part 2: Transform data

processMr() is a very simple function that helps to extract and transform the data into the relevant fields.

const processMr = (mr) => {
  const updatedOn = new Date(Date.parse(mr.updated_at)).toDateString();
  const reviewers = mr.labels.filter(x => x !== 'Review Me');
  const props = {
    mergeRequestName: mr.title,
    mergeRequestUrl: mr.web_url,
    // ...
  };
  return props;
};

For batch process design, it's important to keep your processors decoupled so that it's easy to switch them out when required. Imagine the power you could wield if you had a collection of processors that you can combine and tear apart as you wish.

Part 3: Send notification

Slack uses webhooks for posting to channels. Given the processed data, it's simple to build a POST request according to Slack's block kit design.

fetch(`${SLACK_WEBHOOK_URL}`, {
  body: JSON.stringify(message),
  ...slackPostOptions
});

However, the troublesome part is actually building the payload itself. Would not go into details as it's dependent on Slack's documentation, but it's a fun exercise in composing JSON objects.

Example payload

{
  blocks: [
    { type: 'section', text: [Object] },
    { type: 'section', text: [Object] },
    { type: 'section', text: [Object] }
  ]
}

Deploying it

Nearly all of our services are hosted on AWS EKS (Kubernetes). So we have to dockerize this service, which is as simple as this.

FROM node:lts-alpine as base

RUN apk update --no-cache \
  && apk upgrade --no-cache
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
ENV TZ="UTC"
COPY . .

CMD ["node", "app.js"]

This goes in as a cronjob for our EKS. It's set to run every weekday at 1pm.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  # ...
spec:
  schedule: 0 5 * * 1-5
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 2
  successfulJobsHistoryLimit: 3
  jobTemplate:
    metadata:
      # ...
    spec:
      backoffLimit: 2
      template:
        metadata:
          # ...
        spec:
          restartPolicy: Never
          imagePullSecrets:
            # ...
          containers:
            - name: gitlab-slackbot
              image: mycfsg/gitlab-slackbot:latest
              imagePullPolicy: Always
              env:
                - name: GITLAB_URL
                  value: <URL>
                - name: GITLAB_PROJECT_ID_LIST
                  value: "[<ID_LIST>]"
                - name: GITLAB_MR_OPTIONS
                  value: state=opened&labels=Review+Me
                - name: GITLAB_MR_REVIEWERS_NUM
                  value: "2"
                - name: SLACK_WEBHOOK_URL
                  value: <WEBHOOK_URL>
                - name: GITLAB_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: gitlab-slackbot
                      key: GITLAB_TOKEN
              resources:
                limits:
                  cpu: 150m
                  memory: 150Mi
                requests:
                  cpu: 100m
                  memory: 100Mi

What does it look like?

There are two types of design, one where it includes more details, and a compact one that gets straight to the point. It's just a POC/MVP at this point and it will be further refined based on feedback from the team.

Normal design

Compact design

Moving on

Remember what I mentioned earlier about how the MR process is just one of the methods of maintaining high quality codebases?

So what happens after the MR is approved? For that, we have something called the "chicken" process to help us with having a more stable pipeline.

The "Chicken" Process: How we tackled a lack of Quality Ownership

Sze Ying 🌻 ・ May 16 ・ 6 min read

#productivity #agile #quality #codequality

extra notes:

MR growing because our team is growing in size
This is really more of a glorified reminder than a bot

Automating Sprint Review Slides with Pandoc

Dickson Tan — Fri, 12 Jun 2020 13:51:45 +0000

In the MyCareersFuture team, we conduct sprint reviews at the end of the sprint to showcase what the team has achieved. We use PowerPoint slides to highlight stories that have been done and those which are still being worked on. Preparing this by hand was time-consuming. Here's how we automated the process.

Getting Story Data
Generating PowerPoint Slides
Customizing Appearance of Generated Slides
Limitations
Conclusion

Getting Story Data

We use Pivotal Tracker for project management. The Pivotal Tracker API returns stories in the current iteration or sprint via the /projects/{project_id}/iterations?scope=current_backlog&limit=1 API call. The response looks like this:

{
  "finish": "2020-05-19T00:00:00Z",
  "kind": "iteration",
  "start": "2020-05-04T00:00:00Z",
  "stories": [
    {
      "kind": "story",
      "id": 563,
      "created_at": "2020-05-19T12:01:00Z",
      "updated_at": "2020-05-19T12:01:00Z",
      "estimate": 3,
      "story_type": "feature",
      "name": "Complete construction of the Expeditionary Battle Planetoid",
      "description": "Palpatine was impressed with the PoC, make this one bigger",
      "current_state": "accepted",
      // lots more data for this story
    },
    // data for other stories in that sprint
  ]
}

Generating PowerPoint Slides

Programmatically generating a PowerPoint file isn't for the faint of heart even in a language that has a library for the task. This is where Pandoc comes to the rescue.

Pandoc is an open source tool that converts between almost any markup format that you can think of, including from markdown to PowerPoint. By using the Pivotal API, I wrote a tool that outputs markdown source which Pandoc then converts into the actual slides. Here's what the markdown might look like:

---
title: "Sprint 63"
date: 18 Sep - 02 Oct
---

## Sprint backlog - Done!

*Jobseeker*

* Create a service/API to serve constants to all other modules (5 points)
* change redux code to Typescript for ui-jobseeker
* Populate the content placeholder on the category landing page with the html content (batch 2)
* Implement page title and meta description for category landing page (batch 2)
* Revise content piece, page title and meta description for Risk Management category landing page 

*Employer*

* Add "Employers Toolkit" link in Employer navbar

## Jobseeker - In Progress Features

*Delivered*

* Implement VDP link for MCF jobseeker and make changes to footer

*Finished*

* Implement GA tracking on the recommended jobs on the application confirmation page
* Switch to MCF figures for application count of all jobs
* Include "Search term" in recommender API
* Implement recommended jobs on JD page (getting the jobs from DSAID API)
* Implement data science tracking for view JD page
* Set up company profile page
* Implement data science tracking for the jobs returned on each page of the search page

*Started*

* Change copy for government scheme section on JD page and search results page
* Implement recommended jobs on JD page (frontend)
* Company profile page - jobs section
* Company profile page - About the company section
* Company profile page meta description
* Take contact number and email address from MCF instead of MySF 

## Jobseeker - In Progress Chores

*Started*

* Improve stability of job application tests
* Jobseeker: Cypress Tests for GA calls

Saving that to a file named presentation.md, the following Pandoc invocation generates presentation.pptx:

pandoc presentation.md -o presentation.pptx --self-contained --reference-doc=template.pptx --slide-level 2

When there is too much content to fit into a slide or if ad hoc changes need to be made, the generated markdown and PowerPoint file can be edited afterwards.

The references to "Jobseeker" and "Employer" are the 2 squads in MCF which implement the jobseeker and employer facing portions of the product respectively.

Here's how markdown's semantics naturally map to slides:

Lines 2 and 3 in the source above specify metadata about the document, title and date in this case. This is part of Pandoc's support for front matter and causes a title slide to be generated with this information.
Pandoc was configured to use a slide level of 2. Hence, the level 2 headings are used as slide titles of new slides.
Text between level 2 headings appear on slides as content. Any markdown formatting there such as lists and emphasis causes the corresponding formatting changes in the output.
A line with 3 consecutive dashes (---) causes a new slide to begin after that point. This isn't used in the example above but is very useful when content needs to be split between slides without creating a title for each slide. Refer to Pandoc's user guide for other directives to insert pauses, incremental lists and speaker notes.

Customizing Appearance of Generated Slides

The --reference-doc=template.pptx argument instructs Pandoc to use a PowerPoint template which provides some control over layout and formatting. Although Pandoc technically supports using your own templates, Pandoc crashed often when I tried doing so¹. Hence, I recommend exporting a copy of Pandoc's default PowerPoint template and using it as a starting point for customization with the following command:

# writes Pandoc's default PowerPoint template to template.pptx
pandoc -o template.pptx --print-default-data-file reference.pptx

Open this file in PowerPoint and under the view ribbon, activate slide master view.

The first slide is the master slide. Changing the formatting of the title and text at various list levels applies the corresponding formatting when used to generate slides, a poor man's CSS.

Here's the Slide Master for our sprint review slides set to use the Arial font instead of the default Calibri. Notice how this matches with the content slides above.

Similarly, here's our Title Slide layout.

Limitations

The format of content was kept as simple as possible - essentially a title and content in lists. Pandoc's PowerPoint writer technically supports a 2-column layout and the use of tables. However, we ran into issues because we could not adequately customize the formatting applied to our needs. For instance, each column in a table could not have different widths causing text to often go offscreen.

Besides PowerPoint, Pandoc also supports creating PDF slides via LaTeX and HTML5 slides via Slideous, Slidy, DZSlides or reveal.js. These formats allow virtually full customization of the generated slides, including the use of JavaScript and CSS for HTML5 presentations.

Conclusion

We've been automating most of the preparation for our sprint review slides this way for almost a year now, saving an hour each sprint. If you're preparing slides with information from a system that exposes an API or prefer the text editor like I do, I highly recommend trying Pandoc out.

this seems to happen if the template file is not compatible with the set of XML directives Pandoc recognizes for PowerPoint files. For example, Pandoc wouldn't accept the template file from a set of slides prepared with Google Slides. ↩

The "Chicken" Process: How we tackled a lack of Quality Ownership

Sze Ying 🌻 — Sat, 16 May 2020 09:38:46 +0000

A few months ago, we noticed that our sprint spillovers were constantly increasing in size. Upon taking a closer look, we realised that our existing workflow had an unhealthy emphasis on the number of merges into master. This emphasis was causing a false sense of development efficiency while introducing inefficiency in our delivery process.

The Problem: We left Quality Ownership to our Quality Engineer (QE)

A closer look at our agile workflow signalled that our Quality ownership was not evenly distributed.

	Delivery Stage	People involved
1	Story refinement and prioritisation	Product Owners, Devs, QE
2	Development of story on a new Merge Request (MR)	Devs
3	Code reviews	Devs, QE
4	Merge story into master branch	Devs
5	Ensure master pipeline runs to completion for story	QE

We use GitLab's CI service for our automated pipeline, which runs on every new commit to the master branch. It runs our unit and integration tests before deploying the new build to our QA and UAT environments.

In an ideal world, this would be the perfect setup — after all, everything was ~automated~. But here comes the problem: our integration tests were not stable. With this workflow, the burden fell on our QE to investigate and resolve each integration test failure. This task became exponentially harder with multiple MRs being merged at once by different devs, as well as the existence of undocumented flaky tests. It also didn't help that at that time, our devs to QE ratio was 5:1.

So many merges, yet not one green build...

This caused a bottleneck in our delivery process, which became especially visible at the start of each sprint, when we looked at the spillover from the previous sprint. Our spillover kept growing with majority of the stories stuck in the testing phase, but since the devs had "finished" all our work in the previous sprint, we continued to take up new stories.

We were now taking up new stories every sprint, but they were not being delivered to production at the same rate to create value for our users.

The "Chicken" process

Our old process went on for a while, frequently causing us to go weeks at once without a single green build. After many sprints of digging through numerous merges to debug pipeline failures, Jin Jie (the pained QE mentioned above) and Dickson (a developer who was helping our poor QE) from our team came up with the "Chicken" process. These were the rules for the new process:

One Merge at a time — Only one MR should be merged into master at any time, with the author of this MR being the "Chicken".
The "Chicken" owns the pipeline — The "Chicken" has to ensure that our CI pipeline runs to completion with a green build for their new merge. If there are any genuine integration test failures due to the new merge, open new MRs to fix them. The "Chicken" is free to open and merge as many MRs needed for a green build.
Ownership lies with the "Chicken", but Responsibility is shared by the team — The "Chicken" is not expected to fix all pipeline failures by themselves. If there are too many integration test failures, the "Chicken" is free to ask for help from the rest of the team.

Fun fact: The "Chicken" process was named after our residential chicken squeeze toy, which was used to get everyone's attention in the past. These days, it is a real life semaphore for merging into our master branch!

We have been trying out this new process for a few months now, have observed some significant benefits.

Quality is now a Shared Responsibility

Essentially, the goal of this new process is to eliminate the Somebody Else's Problem effect.

"The Somebody Else's Problem (SEP) field... relies on people's natural predisposition not to see anything they don't want to, weren't expecting, or can't explain"

— Douglas Adams in Life, the Universe and Everything

The "Chicken" process helped to mitigate the SEP field in our team with the following:

Greater visibility on our processes — Now that the definition of "Finish" includes ensuring that the corresponding merge passes the full pipeline, we are more inclined to learn what happens after merging our MRs into the master branch.
More context of the codebase — Investigating the codebase to fix tests gave us the opportunity to learn more about some of the long-lost contextual knowledge behind the implementation of older components.

Rather than having a single QE shoulder the quality burden, quality ownership was now a shared responsibility.

Greater Code Standards

We also saw an improvement in our code standards:

Greater focus on code quality — Having to experience the pain of investigating integration test failures, we have become more mindful of whether each MR we author will break existing integration tests. The old "merge and forget" mentality was now thrown out of the window; there is greater focus on writing reliable code and tests.
Discovering old bugs hidden in the codebase — With more effort in investigating test failures, we have also uncovered several bugs that were not caught during the development and review processes.
Fewer flaky tests — We spent a huge amount of time in investigating some test failures, only to realise that they were flaky. In other words, these tests would only sometimes fail, and the failures were largely due to the choice of testing strategy rather than incorrect implementation. This has motivated the team to work together to improve the overall reliability of our tests, rather than just clicking "retry" N times.

Improved Delivery Efficiency

Before the "chicken" process, we used to see 1-2 green builds per two-week sprint. Now, we are seeing 1-2 green builds every two days. On good days, we might even get up to 3 green builds in a day! 🚀 This was all possible due to

Resource reallocation to remove bottleneck — Rather than one QE investigating test failures for every single merge, the whole team was now invested to do so together. This allowed us to deliver stories at a rate closer to the rate at which we pick them up.
Debugging is easier when the diff is smaller — With only one merge going into the master branch at once, it was easier to inspect the diff due to the new merge. This made it a lot quicker to pinpoint where the failure point was when the pipeline failed on a new merge.

Our average number of green builds has increased!

Since adopting this new process, we have also made several modifications to it. One example is allowing multiple merges at once as long as there is still at least one person appointed as the "chicken". This modification helped us reap the benefits of the "chicken" process while also catering to our larger team size. We hope to continue improving on this process as our team's needs evolve.

Thank you Jin Jie and Dickson for your great efforts in improving our processes and also Joseph for providing feedback on this piece! Also thank you Jin Jie for providing the diagrams.

If you are now intrigued to work in a team like ours: we are hiring! Drop me an email @ ting_sze_ying@tech.gov.sg if you are interested to find out more 🌈

Run remote retrospectives using Trello

Andrey Bodoev — Thu, 07 May 2020 11:39:05 +0000

Here's quick walkthrough how we run retrospectives using Trello boards. It's fairly simple workflow, and can be adapted with easy.

Few things to add,

this approach turns out works well for both remote and in office environments;
can be highly customized depends on needs;
I haven't say that in video, but it's removes frictions with post-its, and use cards instead, and creates a possibilities to have issues to discuss in advance, prior to retrospective.

I'm using Playbook framework by Atlassian as an example for this board

If you have any questions, please let me know in comments below.

Hidden power of Commit Guidelines

Andrey Bodoev — Sun, 03 May 2020 10:40:47 +0000

In our projects we adapted Commit Guidelines, with more or less standard variation of Angular Commit Guidelines. It deliver what it promise,

This leads to more readable messages that are easy to follow when looking through the project history. But also, we use the git commit messages to generate the change log.

Since we adapt such guidelines, I have discovered one powerful effect on a developer growth.

It's how you start thinking about code changes you commit to code base. Following questions starts to bubbling inside your head,

Does this change belong to this commit?
What's clear intent of your changes in?
What's reasoning and thoughts I can put in message?

And etc.

What happens now is that each commit is representing some type of change, with describing clear intent encapsulated inside of commit message.

Suddenly, you start reading git log (yes, for real), and if you need to do comparison between log histories you can do this by simply looking at the titles of commit messages.

Here's one example,

$ git log --pretty="%n    %s" --name-only

    test: has Cancel button, to check both confirm branches

cypress/integration/FunctionalTesting_Suite/CompanyProfilePage/company_profile_page.spec.js

    refactor: move Cancel button to CompanyProfile components

src/components/CompanyProfile/CancelButtonWithConfirmation.scss
src/components/CompanyProfile/CancelButtonWithConfirmation.tsx
src/components/CompanyProfile/CancelButtonWithConfirmationContainer.tsx
src/pages/CompanyProfile/CompanyProfile.tsx

    feat: Employer - Company profile page Cancel button

src/pages/CompanyProfile/CancelButtonWithConfirmation.scss
src/pages/CompanyProfile/CancelButtonWithConfirmation.tsx
src/pages/CompanyProfile/CancelButtonWithConfirmationContainer.tsx
src/pages/CompanyProfile/CompanyProfile.tsx

In git log you can tell, that I finished feature, did some refactoring, and added integration tests afterwards. That was thoughtful workflow.

Can you tell the same story, by looking at example below?

$ git log --pretty="%n    %s" --name-only

    Changed scss

src/pages/CompanyProfile/CancelButtonWithConfirmation.scss

    Add Cancel button

src/pages/CompanyProfile/CancelButtonWithConfirmation.scss

    OK it doesn't work, forgot component. LOL

src/pages/CompanyProfile/CancelButtonWithConfirmation.scss
src/pages/CompanyProfile/CancelButtonWithConfirmation.tsx

    Tests

src/pages/CompanyProfile/CancelButtonWithConfirmationContainer.tsx
src/pages/CompanyProfile/CompanyProfile.tsx
cypress/integration/FunctionalTesting_Suite/CompanyProfilePage/company_profile_page.spec.js

    Is it working yet?

src/pages/CompanyProfile/CompanyProfile.tsx

Of course, you may not look at this without crying. You may end up, shamefully do git rebase to squash your commits to hide your crimes of uncertainty.

To start adopting commit guidelines, I would recommend to look at this project http://commitizen.github.io/cz-cli/

Code Review: Prereview checklist. No important tasks are forgotten.

Andrey Bodoev — Sun, 26 Apr 2020 06:26:36 +0000

Here's my rephrase on preflight checklist,

a prereview checklist is a list of tasks that should be performed by developer or team prior to raise an Merge Request

Everytime we discussing changes in code review, we might found ourself talking about same things over and over again. For example, asking in comments to add testing instructions or adding pivotal story number for reference.

To prevent that we need a little reminder to ourselves, or list of things we should check before raising an attention to your changes.

In our team we have checklists built-in in our Merge Request templates.

Here an example of one of our templates,

<!-- Make sure Merge Request title following "<type>: <subject> [#<pivotal_story_id>]" -->

<!-- Brief description. Examples are, screenshots, steps to reproduce, links to dependent MRs -->

### General Checklist:

- [ ] Exact versions in package.json
- [ ] Testing instructions?
- [ ] Docs? Examples are, update `README.md` file, or add ADR in `doc/adr`
- [ ] Tests?
- [ ] Put in copy at least two potential reviewers

/cc

/label ~"Review Me"

Structure of this template are self-explanatory, one thing I want to point out is last line: /label ~"Review Me" on which I already wrote an article.

And don't forget to treat checklists as living documentation, new items can be added, or removed throughout time.

It's purpose is to improve code review safety by ensuring that no important tasks are forgotten.

How we use Google Sheets API to Manage Our Notification Banners

Joseph Matthias Goh — Fri, 24 Apr 2020 10:52:39 +0000

If you've visited our site before, you'd have noticed that banner at the top of our page:

Or you might have noticed the inline banner linking to the #SGUnitedJobs virtual career fair site:

There was a time when we didn't have any of these, so what happened?

The Problem Domain

The header banner has been in-place for some time now, and it was created due to some needs:

We wanted a way to notify users about new features we've released
We wanted a way to notify users of scheduled downtimes because of systems that we're integrating with

More recently, to support whole-of-government efforts to reduce the impact on our workforce, the agency we're servicing - Workforce Singapore - decided to launch an #SGUnitedJobs virtual career fair. And there was a slight issue.

In the media release, the press were quoted the words "MyCareersFuture". This led to the general public landing on our site instead of the virtual career fair's site, resulting in a need to redirect users to the correct site.

This meant some updates to our landing page notification/updates elements:

The header banner needed to be clickable
An in-page element needed to be created

In addition to addressing users' problems, our solution also needed to accomodate use cases by our business elements, and they needed to be able to enable text-changes to be made without a new system deployment.

The Solution

We begin with the end.

Google sheets for us was a nice interface between business and technical elements. Business users could enter in information as it came along without interfering with the code-level development using a spreadsheet, with the spreadsheet being able to run custom-code to generate a JSON output that our system could consume.

"Wait, can Google sheets do that?" I hear you ask.

Setting up Google Sheets as a JSON API

Let's set up a simple Google Sheets that you can use as an API that's similar in nature to what we've done.

In MCF's backend, we consume data from Google Sheets, run validations on it before throwing it back out as a .json file which we push to our CDNs so that these can be consumed by our users without placing a load on our servers.

1. Create a new spreadsheet

Go to https://drive.google.com and create a new spreadsheet. Maybe create a table like:

ID	Name	Username	Email
1	Joseph	joseph	j@seph.com
2	Matthias	matthias	m@tthias.com
3	Goh	goh	g@h.com

PROTIP: The table above works like copypasta with Google Sheets if you copy it properly

2. Publish to the web

Go to the top navigation bar, and access File > Publish to the web. Confirm that Link is selected and select Sheet1. Change the type to Comma-separated values (.csv). Click the Publish button and say OK.

A link should be provided to you. Test it out by pasting it into the address bar of your page. A .csv file should be downloaded. Opening it up should reveal (if you've entered in the information as-if from above):

ID,Name,Username,Email
1,Joseph,joseph,j@seph.com
2,Matthias,matthias,m@tthias.com
3,Goh,goh,g@h.com

3. Making it JSON

We'll be inserting the JSON as values in the empty JSON Output column you've copied above using the Script Editor. In the header navigation menu, go to Tools* > **Script editor. A new Code.gs should be waiting for you.

Overwrite the generated code and paste in the following code:

function onEdit(e){
  const editedRow = e.range.getRow();
  const activeSheet = SpreadsheetApp.getActiveSheet()
  const sheetName = activeSheet.getName();
  switch (sheetName) {
    case "Sheet1":
      const jsonData = createUserJSON(activeSheet, editedRow);
      const cell = activeSheet.getRange(editedRow, 5);
      cell.setValue(jsonData);
  } 
}

function createUserJSON(sheet, editedRow) { 
  const userJSON = {
    id: sheet.getRange(editedRow, 1).getValue(),
    name: sheet.getRange(editedRow, 2).getValue(),
    username: sheet.getRange(editedRow, 3).getValue(),
    email: sheet.getRange(editedRow, 4).getValue(),
  }
  return JSON.stringify(userJSON);
}

Google Script is essentially JavaScript. Also, if you've changed your sheet name, you might want to replace the "Sheet1" in the switch-case branch to the name of your sheet.

The above code is triggered on an edit to the sheet. It checks whether the edited sheet is named Sheet1 and if it is, generates a JSON string and pastes it into the 5th column of the row being edited. Save it by going to File > Save. The name shouldn't matter.

Go back to your sheets and edit one of the existing values. On removing your focus from that cell, you should see a JSON value appear in the 5th column.

4. Consuming the JSON

Retrieve the link from the Publish to the web stage. We're going to use that to retrieve our JSON outputs. Your original link should look like:

https://docs.google.com/spreadsheets/d/e/2PACX-XXX/pub?gid=YYY&single=true&output=csv

Confirm you've done an edit so that the JSON appears and then do a curl to see what it looks like now. In your terminal:

curl 'https://docs.google.com/spreadsheets/d/e/2PACX-XXX/pub?gid=YYY&single=true&output=csv'

You should receive something similar to the following as your response:

ID,Name,Username,Email,JSON Output
1,Joseph,joseph,j@seph.com,"{""id"":1,""name"":""Joseph"",""username"":""joseph"",""password"":""j@seph.com""}"
2,Matthias,matthias,m@tthias.com,
3,Goh,goh,g@h.com,

We're almost there!

5. Tying It All Up

Now that we can generate the JSON, go ahead and update the other rows so that all of them have a value in the JSON Output column.

In this step, we'll link all the JSON Output values into a new sheet which we can use as the data source in a service.

Add a New Sheet by clicking the plus (+) symbol at the bottom left of the existing spreadsheet. Name it Output. We'll be writing the aggregated JSON Output to row 1 column 1 of this sheet.

Go to Tools > Script editor once again and modify the script there so that it looks like:

function onEdit(e){
  const editedRow = e.range.getRow();
  const activeSheet = SpreadsheetApp.getActiveSheet()
  const sheetName = activeSheet.getName();
  switch (sheetName) {
    case "Sheet1":
      const jsonData = createUserJSON(activeSheet, editedRow);
      const cell = activeSheet.getRange(editedRow, 5);
      cell.setValue(jsonData);
  }

  // > diff starts here
  const jsonOutputs = activeSheet
    .getRange("E2:E")
    .getValues()
    .filter((val) => val[0].length !== 0);
  const aggergatedUsers = [];
  jsonOutputs.forEach(function(value) {
    const user = value[0];
    aggergatedUsers.push(JSON.parse(user));
  });
  const outputSheet = SpreadsheetApp.getActive().getSheetByName('Output');
  const outputCell = outputSheet.getRange(1, 1);
  outputCell.setValue(JSON.stringify(aggergatedUsers, null, 2));
  // / diff ends here
}

function createUserJSON(sheet, editedRow) { 
  const userJSON = {
    id: sheet.getRange(editedRow, 1).getValue(),
    name: sheet.getRange(editedRow, 2).getValue(),
    username: sheet.getRange(editedRow, 3).getValue(),
    email: sheet.getRange(editedRow, 4).getValue(),
  }
  return JSON.stringify(userJSON);
}

What we just added basically retrieves the value of all rows in column E which in the sample sheet we created corresponds to the column JSON Output. It then transforms the retrieved values into an array object before converting it back into a string using JSON.stringify. The null, 2 arguments in the JSON.stringify call basically indicates to format the JSON in a human readable way (see JSON.stringify documentation on MDN if you're interested in what exactly it does)

After you've made the change, save the GoogleScripts project, head back to your sheet and go ahead make a modification to one of the columns to trigger an output to the Output sheet in your spreadsheet.

Go to File > Publish to the web once again, this time selecting the Output sheet and selecting Tab-separated values (.tsv) as the type. In the Published content and settings section, ensure that Output is also being published and that the checkbox with Automatically republish when changes are made is also checked.

Copy the provided link which look like (sensitive values are masked with XXX)

https://docs.google.com/spreadsheets/d/e/2PACX-XXX/pub?gid=YYY&single=true&output=tsv

To modify this such that it will always return just the first row and column, append a &range=A1 so that the final link looks like:

https://docs.google.com/spreadsheets/d/e/2PACX-XXX/pub?gid=YYY&single=true&output=tsv&range=A1

Let's test this out by running a curl with it which would be what your Request module would be doing if it calls this URL:

curl -vv 'https://docs.google.com/spreadsheets/d/e/2PACX-XXX/pub?gid=YYY&single=true&output=tsv&range=A1'

To test that it's valid JSON, we can pipe it to a tool called jq that can help us validate it:

curl -vv 'https://docs.google.com/spreadsheets/d/e/2PACX-XXX/pub?gid=YYY&single=true&output=tsv&range=A1' | jq '.'

We're done. Aren't we?

6. What if something goes wrong?

As with all code, things can and will go wrong during development, but the awesome news is that Google provides us with a page where we can view errors in our script and this can be found on the Code.gs page through the file navigation menu via View > Executions which should open the script's Google Apps Script dashboard.

Note that only calls to console.error go through to this dashboard.

Looks like we're done for real!

If you like what you just read, don't forget to leave some reactions/comments so we know this has been interesting for you- and do consider following us for more insights into tools we use and our development processes.

Cheers and till next time!

Code Review: Review labels in GitLab

Andrey Bodoev — Sun, 19 Apr 2020 12:27:24 +0000

We use GitLab to host our code. One of the aspects of it is deal with code reviews on daily basis.

In our team, we're about 10+ devs, it's quite difficult to have someone attention to code review, if it's product team and most of the time you focus on your own work and hoping if someone is review your changes.

Initially, we have this,

Here's an issue, which is no way to tell if someone is reviewing your code, or if you have enough reviewers and etc. Number of comments are not gave you much, and 0 of 2 is not really helpful here.

In our code review guidelines we have few rules, one of them is to have your Merge Request (abbr. MR) to be reviewed by at least 2 person in team.

There's few concerns to address,

how to make it visible if someone is reviewing your MR;
who reviewing what, and have they done so;
do you have enough reviewers.

Gitlab has it's own approval system in place, and code review guidelines. It does work, but we decided to tweak this a bit to improve communication within a team.

There's few types of labels we introduces,

Review Me, label is used to have an attention to MR and stating that it's ready to be reviewed, otherwise it can be wip;
Review by Paul Atreides, each reviewer has it's own label assigned once they start commenting;
Approved by Padishah Emperor Shaddam IV, each reviewer has it's own label once review is done, and comments are addressed.

With this in practice, once you going to visit list of MRs waiting. You're not checking each of them in order to find out, if it's been reviewed by someone else and it's ready to be merged. And it's greatly improve visibility of state on each Merge Request.

Code Review: Name file after exposed function

Andrey Bodoev — Fri, 10 Apr 2020 11:15:24 +0000

In code reviews which we conduct in our team, I might find some examples of code, which is can be misleading or with no clear intent. This post is about to share reasoning and hear critique on some of these examples.

Name file after exposed function

One of the cases is having few functions in one file, related or maybe not, or can be completely not related, but somehow group in one file.

get/
  index.ts
    export getSomething :: Int -> Int
    export getSomethingElse :: String -> String

Few things we can tell immediately,

please don't use index.js, https://www.youtube.com/watch?v=M3BM9TB-8yA&vl=en
both get directory and index.ts gave no context whatsoever.

Instead, try to name files after exposed function, and move each of them into it's own file, like so

getSomething.ts
  export getSomething :: Int -> Int

getSomethingElse.ts
  export getSomethingElse :: String -> String

It shows intent by looking at filename.

Few side effects which you may discover later,

Functions are enclosed in file to ensure there's no shared variables; which leads to proper unit tests and maintainability.
And after this change, there's powerful way to quickly go through project structure; by opening files which named after function and not searching through functions in search output of your text editor.

Takeaways

Name file after exposed function;
Breaking up into small modules for unit testing and maintainability;
Browsing code within file vs. browsing code within project.

One of the great examples in wild, https://github.com/lodash/lodash/tree/master/

DEV Community: MyCareersFuture

Setting up ZSH on Android

What you need

Steps 🚶

My setup 🤓

My use case 🦄

Getting started with grafana and prometheus for kubernetes metrics

Table of contents

Requirements

The quick 5-minute install

What you get

Setup

Storage space

Installation

Connecting Grafana to Prometheus

Adding Dashboards

How does it work?

Node Metrics

Application Metrics

So how is this configured?

Wrapping up

Getting started with deploying grafana and cloudwatch metric dashboards

The Pillar of Metrics

Table of contents

Requirements

The quick 5-minute build

What you get

Setup

ryanoolala / recipes

A collection of recipes for setting up observability toolings

recipes

Requirements

Cloudwatch IAM Role

Grafana

Installing Grafana

Show me the UI!

Whats next?

The 10-minute build

What you get

Setup

Postgres RDS

Grafana

Logging in

Wrapping up

Gitlab MR Bot: Getting people to do code reviews

MR reviews in MCF

Why aren't more people reviewing?

Temporary solution

Introducing the MR Bot

Part 1: Fetch data

Part 2: Transform data

Part 3: Send notification

Deploying it

What does it look like?

Normal design

Compact design

Moving on

The "Chicken" Process: How we tackled a lack of Quality Ownership

Sze Ying 🌻 ・ May 16 ・ 6 min read

Automating Sprint Review Slides with Pandoc

Getting Story Data

Generating PowerPoint Slides

Customizing Appearance of Generated Slides

Limitations

Conclusion

The "Chicken" Process: How we tackled a lack of Quality Ownership

The Problem: We left Quality Ownership to our Quality Engineer (QE)

The "Chicken" process

Quality is now a Shared Responsibility

Greater Code Standards

Improved Delivery Efficiency

Run remote retrospectives using Trello

Hidden power of Commit Guidelines

Code Review: Prereview checklist. No important tasks are forgotten.

How we use Google Sheets API to Manage Our Notification Banners

The Problem Domain

The Solution

Setting up Google Sheets as a JSON API

1. Create a new spreadsheet

2. Publish to the web