DEV Community: Kelvin Tay

Penciling career goals for 2024

Kelvin Tay — Sat, 30 Dec 2023 23:15:07 +0000

It’s the New Year’s Eve morning here in Tokyo, Japan. I’m typing this while cradling the baby at a Starbucks. Staring at the morning rain, I thought it’s a good time to reflect on 2023 while welcoming 2024.

Gratitude

Flexibility in a remote-first company. Laundry, accompanying the wife for her doctor appointments, etc. Thankful for the team’s understanding.
3-month parental leave. It takes a village to raise a child. I think it would be too difficult for just one parent to navigate the first 3 months alone.
Promotion. The new role encourages me to continue/start mentoring, glue work, technical leadership.

Failings

Acted on things when I was not the best candidate to. I need to be be more aware of saying “no” to others and myself. Sometimes, you can become less helpful with your answers.
Did not listen to the body. Our bodies will likely signal to us when we are overworked and stressed. Thankfully, it is better now, as I juggle priorities (family) but I wish I was better at this from the start.

Hey 2024

Greater technical depth in the devops space. Metrics, k8s, networking.
Champion others; would love to help others step up / promote.

if you stumble into this post, I hope you and your family enjoy a good 2024 ahead!

Best wishes!

buy Kelvin a cup of coffee ☕

Reflections: From Software Engineering to Technical Support

Kelvin Tay — Thu, 19 Oct 2023 01:50:57 +0000

After 8+ years as a software engineer, I made a career switch in 2021. I joined CircleCI as a support engineer.

As I enter my 3rd year with CircleCI, I thought it may be good to reflect on this career change so far.

I noticed there are many literatures for support engineers looking to switch to software engineering / development. However, there aren't many opinions on the other direction.

I hope this can be helpful for folks deliberating on similar career changes.

Why I switched

I will be honest here. One of the major reason I switched was due to the salary compensation.

I was drawing a comfortable salary at my previous job. However, my wife and I were hoping to start a family and consider purchasing a home. I was able to negotiate for a much-higher salary in this switch. (I know this is likely a rare case in many countries. For context, I am based in Tokyo, Japan.)

I also wanted to try something (slightly) different.

Covid-19 was much less of a pandemic by 2021, but it prompted me to relook at life and its (un)certainties. One of the interviewers asked me “where do you see yourself in 3 years?”. I took the opposite view and considered what happened in the last 2 years, from 2019 to 2021. So much has changed across societies as we battled Covid-19. For me, the unexpected Covid-19 did spur me to try something different. Life is too short to be doing the same thing for a lifetime.

Critically, I also enjoyed using CircleCI as a customer. Don't get me wrong; My past adventures were with companies I did believe in. However, being a heavy and happy customer of CircleCI, I found it empowering to be in a position to share my enthusiasm with the customers.

How has the journey been

Fortunately, CircleCI is a product for software teams. This meant that, as a support engineer, I am tackling problems with another software engineer on the other side. I can empathize with the customer.

I have found it really rewarding to be able to draw on my past experience, and share best practices and advice with the customers.

Being part of the JAPAC (Japan + Asia Pacific) team, I also get opportunities to brush up my Japanese. To be honest, my wife and Deepl helped me a lot. I never aspire to reach near-native level for Japanese, but this has been a positive bonus.

One of the major perks is also learning new technologies. I get to learn from both our engineers, and also our customers (e.g., their tech stack). Prior to this job, I've never dug deep with Packer, Nomad, Git LFS, and even Windows PowerShell to name a few. (You can find my public repos for testing here)

Would I recommend it?

In my case, I've enjoyed the transition so far. Thankfully, I still get to utilize much of my software engineering knowledge to troubleshoot with customers.

However, like any choice made, technical support as a career comes with challenges too.

I think being able to empathize with customers is critical. Beyond the technical know-how, you'd need to know how to handle delicate communications at times (e.g., system-wide incident).

If you are a software engineer reading this, I would encourage you to consider the switch if:

(Traits)

You enjoy communicating with people as much as digging deep into code / configurations.
You enjoy debugging, or chasing down root causes as much as implementing new features.

(Strengths)

You can empathize with users, and can advocate on their behalf. I think folks contributing to open-source projects may be good candidates.
You are patient and can negotiate tough communications.

(Motivations)

You would like to be closer to customers and the product.
You enjoy opportunities in sharing best practices with customers.
You believe in and enjoy the product.

These are just my opinions based on my experience so far. Your mileage may vary.

However, remember that switching does not mean you cannot go back. You can and should re-evaluate this decision along the way.

buy Kelvin a cup of coffee ☕

Troubleshooting like a Doctor

Kelvin Tay — Thu, 27 Jul 2023 09:31:52 +0000

I had recently been introduced to the SOAP acronym used by medical professional by my brilliant manager.

This was really interesting for me, given that I had no prior “professional” experience or course on advising and troubleshooting.

We can draw many parallels indeed in technical support.

Subjective :

“Well, it worked on my local machine.”
“Our builds started failing from last Wednesday.”

Objective :

Obtaining and investigating build logs
Extracting details from shared screenshots
Verifying dependent external services are/were not down

Assessment :

Failed builds relied on a remote service which had recently introduced breaking changes. Likely, the customer ran on their local machine before the change was introduced.

Plan :

Share assessment on remote service, and confirm if the customer's reruns on local machine fail now with the same error.

Proceed to share workaround or solution if available.

buy Kelvin a cup of coffee ☕

How I added CircleCI Runner support for Terraform provider

Kelvin Tay — Sun, 25 Jun 2023 02:40:19 +0000

I have released support for the CircleCI Runner resource-class and token in my unofficial Terraform provider for CircleCI, as per v0.10.3.

Developers can now manage the provisioning (and teardown) of CircleCI self-hosted Runners within Terraform. You can explore an example here.

This was a fun challenge, and I wanted to document my journey on this work.

Investigation

Unlike other resources, self-hosted runners are not manageable under the official V2 API. Developers had to use the CircleCI CLI to manage resource-classes and tokens instead.

To port this into my Terraform provider, I was hoping there was a HTTP API available. This way, I can continue using my approach of abstracting the HTTP API away to a Go SDK.

I assumed (wrongly) the CLI was using GraphQL under the hood for Runner operations, as with many others (e.g., for Orbs).

Digging into the source-code, I then realized Runner resource-classes and tokens can be managed via a HTTP API; It was simply not publicly documented, yet.

Glue

After the legwork mentioned above, I tested the HTTP APIs with an OpenAPI (Swagger 2.0) document. This enabled me to generate a Go SDK for Runner APIs.

Why are the Go SDKs separated? Wouldn't it be easier to keep it all in one?

That was something I mulled over for some time indeed. I have documented my reasons for keeping them separate.

Assembly

With the Go SDK published, I “simply” have to then expose the Runner resource-classes and tokens in the Terraform provider codebase.

The main work was done within a pull request here. This also included acceptance tests, and examples.

Ultimately, I noted self-hosted (machine) runners are also available for CircleCI's Server customers (i.e., self-hosted CircleCI). We would want to extend and ensure this addition can be used by platform teams using CircleCI Server.

It turned out that the Runner API:

is via https://runner.circleci.com for CircleCI Cloud
is via https://YOUR-SELF-HOSTED-DOMAIN for CircleCI Server

Thankfully, this was a quick patch. I was also able to verify this fix against my own CircleCI Server instance.

Things I learnt

This feature was satisfying for me to build, and I had many learning points along the journey.

Read the code: This feature would not have been completed if I did not dig deeper into the publicly-available source code 📖
Keep trying: I am still learning (and failing) at Go. However, I think it is important to keep trying and learning. Keeping this source code open-sourced forces me to keep myself honest too about my lack of knowledge. For fellow engineers out there, let's keep at it! 🤓

#terraform #circleci #runner #go

buy Kelvin a cup of coffee ☕

Understanding the nuances of using a Docker image for your CI/CD job

Kelvin Tay — Sun, 26 Mar 2023 03:31:14 +0000

Many CI/CD providers, like GitHub Actions and CircleCI, offer the options to run your CI/CD job using Docker images today.

This is a useful feature since you can ensure your job is always running with the same pre-installed dependencies. Teams may choose to bring their own Docker image, or use readily-available community images on Docker Hub for instance.

One challenge is that the Docker image in use may not have been intended for use in CI/CD automation. Your team may thus find yourselves trying to debug puzzles like:

I'm pretty sure XYZ is installed. Why does the CI/CD job fail to find XYZ?
Why is the FOOBAR environment variable different from what we have defined in the Docker image?

Docker image 101

When you execute the docker container run ... command, Docker will, by default, run a Docker container as a process based on the ENTRYPOINT and CMD definitions of your image. Docker will also load the environment variables declared in the ENV definitions.

Your ENTRYPOINT and CMD may be defined to run a long-running process (e.g., web application), or a short process (e.g., running Speccy to validate your OpenAPI spec). This will depend on the intended use of your image.

In addition, Docker images will be designed to come with just-enough tools to run its intended purpose. For example, the wework/speccy image understandably does not come installed with git or curl (see Dockerfile).

Docker images may also be published for specific OS architectures only (e.g., linux/amd64). You will want to confirm which OS and architecture the image can be run on.

These are important contexts , when designing CI/CD jobs using Docker images.

Understanding Docker images for CI/CD

Generally, for CI/CD automation, your job will run a series of shell commands in the build environment.

CI/CD providers like GitLab CI and CircleCI achieve this by override your Docker image's entrypoint with/bin/sh or /bin/bash when executing them as containers.

This is why you would want to use the -debug tag variant for Kaniko's Docker image when using in CircleCI for instance.

Additionally, your Docker image may not come with the required tools for your CI/CD automation. For example, you would require git in order to clone the repository as part of the CI/CD job steps.

Debug Cheatsheet

With this information in mind, here is a list of commands you can run locally to debug your chosen Docker image.

# inspect the "built-in" environment variables of an image
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Env"
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "LANG=en_US.UTF-8",
  "PYSPARK_PYTHON=python3",
  "SPARK_HOME=/home/glue_user/spark",
  "SPARK_CONF_DIR=/home/glue_user/spark/conf",
  "PYTHONPATH=/home/glue_user/aws-glue-libs/PyGlue.zip:/home/glue_user/spark/python/lib/py4j-0.10.7-src.zip:/home/glue_user/spark/python/",
  "PYSPARK_PYTHON_DRIVER=python3",
  "HADOOP_CONF_DIR=/home/glue_user/spark/conf"
]

# check the default entrypoint
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Entrypoint"
[
  "bash",
  "-lc"
]

# check the default cmd
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Cmd" 
[
  "pyspark"
]

# check tools installed
$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "git --version"
...
git version 2.37.1

$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "python --version"
...
Python 2.7.18

You can also find an example of this debugging for a CircleCI use-case here:https://github.com/kelvintaywl-cci/docker-executor-explore/blob/main/.circleci/config.yml

#docker #cicd #debug #cheatsheet

buy Kelvin a cup of coffee ☕

Prism: a useful developer tool for OpenAPI specs

Kelvin Tay — Sun, 12 Feb 2023 02:54:58 +0000

If your service exposes a HTTPS API, you may already be documenting it via an OpenAPI spec.

For instance, HSBC provides an OpenAPI spec (read: Swagger) for their Personal Credit Cards API (example).

At a previous career with a company that integrates with financial institutions, we developed the culture to describe our APIs with OpenAPI 3.0.

We had the case where our mobile frontend developers were building the UI features faster than our backend team can develop the API endpoints.

Because we agreed on an API contact using OpenAPI 3.0, we were able to provide mock backend servers for them to test with.

And we did it easily with Stoplight's Prism.

All we needed was to provide our OpenAPI spec, and we had a mock backend ready for our mobile frontend. This unblocked both teams, and we always can rely and refer on our API contract as our agreement terms.

Recently, I have been developing a Terraform Provider. Since the underlying cloud service has a HTTPS REST API, I was able to:

Describe the underlying service's API as a OpenAPI spec
Generate a Go client SDK based on this OPenAPI spec
Mock the service with Prism for some quick acceptance tests

Prism does more than mocking; You can also use it to inspect any discrepancies between your API implementation and the API spec. You can find out more on their GitHub page:https://github.com/stoplightio/prism

Highly recommended for developers who use OpenAPI specs as contracts between components/services!

#openapi #mock #testing #API

buy Kelvin a cup of coffee ☕

Reflecting on HashiCorp Certified: Terraform Associate 002

Kelvin Tay — Sat, 21 Jan 2023 03:56:41 +0000

Background

I passed the HashiCorp Certified: Terraform Associate 002 exam on Jan 14th 2023. This was my first try, fortunately.

In terms of my Terraform experience prior to the exams, I have yet to build any Terraform project at a production level. My recent exposure was mainly around trying to play around with CircleCI’s Nomad client Terraform setup.

Motivations

In my past career, I have used loads of AWS CloudFormation. _Yes, I have had some battle scars re: IaC._I always wanted to learn more about Terraform, given that it supports more than 1 cloud platform.

In particular, I am interested in contributing a Terraform provider for CircleCI. This means creating a provider/plugin for creating CircleCI webhooks / scheduled pipelines via Terraform for example.

This above may likely be a lofty ambition, but I thought I’d hedge this by also trying for the certification. Kill two birds in one quarter, as they say.

Breakdown of Score

Content Area	Score (%)
Understand infrastructure as code (IaC) concepts	100
Understand Terraform’s purpose (vs other IaC)	100
Understand Terraform basics	85
Use the Terraform CLI (outside of core workflow)	100
Interact with Terraform modules	50
Navigate Terraform Workflow	100
Implement and maintain state	75
Read, generate and modify configuration	72
Understand Terraform Cloud and Enterprise capabilities	100

Overall, I scored 84% (70% required).

Word of Advice

If you are not using Terraform on a daily basis, practice Terraform CLI commands definitely!

The exam has a few questions where they may test you to type the code / command required. See sample questions here.

I did have a “doh!” moment, since it wasn’t clear how “complete“ the answer should be.

In particular, one question I got was: How would you reference the vpc_id from this module in the code snippet?

To reference the module output, it would be module.aws_vpc.vpc_id for example. I think the full answer should be: vpc_id = module.aws_vpc.vpc_id.

I had only typed module.aws_vpc.vpc_id due to lack of clarity in the question. I think I likely lost points there!

In terms of studying for it, I highly recommend the following Udemy course + practice exam!

Lastly, there will be a newer exam version (003). I understand this update would remove older questions referring to already-deprecated commands (e.g., taint). So if you can wait, do go for the 003 version instead!

Final Thoughts

It was a fun exam to take, for sure!

Given that it is really about a tool and its ecosystem, I’d highly encourage folks to try out Terraform while practicing for the exam.

Your mileage on the Terraform tool will go a long way if you practice. This way, the exam questions will make way more sense too, as you practice.

buy Kelvin a cup of coffee ☕

empathy in these times

Kelvin Tay — Sun, 11 Dec 2022 06:58:41 +0000

I survived a 17% reduction (layoff) at CircleCI last week. As soon as the CEO letter was published, many recruiters reached out.

I can understand that. However, one email irked me bad, that I had to respond in curt.

Without naming the company and the actual person, here is the snippet in raw form:

We learned about the CircleCI lay-offs on Linkedin via your post and [redacted]'s post. It looks like you are not affected, which is great news, but with such incident striking instability for a previously believed to be COVID-proof and recession-proof business, would it have spiked some of your interests to hear out available jobs in the market?

So feel free to refer us [redacted], any other colleague you may know in CircleCI affected by the lay-offs, or anyone in Japan open for a job change in Software Engineering.

In these times, we see many tech companies reducing their workforce. These are sudden events for people affected; No one wants to be in such positions. As a recruiter, please please please give your words some more thought before pressing “send”.

If your team is looking for amazing engineers and people in Japan and beyond, please check out this link of our CircleCI alumni:https://docs.google.com/spreadsheets/u/0/d/1A_1uk6qP_a0C2bwKcvV7G-meypWCgW-giK9O2UG8p_k/htmlview

buy Kelvin a cup of coffee ☕

Terraforming all the things

Kelvin Tay — Thu, 01 Dec 2022 12:08:27 +0000

Infrastructure as Code (IaC) is not a new concept for many. Prior to joining CircleCI, I have been wrangling with AWS CloudFormation long enough (2 years) to bear some of its pain-points.

Since joining CircleCI, I have been exposed to Terraform through internal and external projects. So far, I really like that Terraform is not tied to specific cloud providers.

I also like that any service can contribute a Terraform provider, thereby allowing users to define your service's resources as code. (Any cloud resource that exposes an CRUD / RESTful API is a good candidates for Terraform.)

Recently, I have been wishing many things to be “Terraform-able”.

As a support engineer, I use Zendesk daily. We create Zendesk macros for canned responses in particular. Overtime, these macros grow, and it may be hard to manage or validate changes to our macros. I do wish Zendesk has a Terraform provider, so we can institute our macros in code.

Recently, I also wished that I could Terraform my resume. Then, I snapped out of it.

When you have a powerful hammer like Terraform, everything looks like nails.

#terraform #infrastructureascode #declarative

buy Kelvin a cup of coffee ☕

Do I continue working at my current job?

Kelvin Tay — Sun, 06 Nov 2022 02:56:35 +0000

Primer

This post reflects my opinions on how I decide to continue staying at the current job, or consider switching.

As such, this is purely from my perspective. You would likely have your own principles and outlook on life and career; If anything, I hope this post motivates you to reflect, rethink, and refresh.

For context, I work a full-time job with no additional side gigs or secondary jobs.

My Guiding Principles

I use the following 3 factors to decide if I should continue with the current job:

Compensation
Job Fulfillment
Work-Life Balance

You likely have your own principles, so you can replace this list with your own!

I find that realistically, most jobs should hit 2 out of the 3 factors at least.

When I find myself seriously contemplating a career change, I like to check this list. If I evaluate that the current opportunity hits only 1 factor (i.e., score of 1/3), it is a good indication that I can do better.

1. Compensation

Without mincing my words, this is just about the main salary. I am not swayed by or overly concerned with stock options, or welfare benefits.

If you are working in Japan, I do encourage you to check out https://opensalary.jp/. This can help you get some idea about what others with similar tech backgrounds are compensated.

2. Job Fulfillment

This can a tough one to judge or measure.

You can go all logical about it (i.e., making a list of the positives and negatives).

Or, you can also simply base it on gut feeling (e.g., say 8 out of 10).

For me, I tend to find my answer by asking myself:

“If someone asks me what I do for a living, how enthusiastic would I be about introducing my current job?”

In terms of motivation, I like to measure how I feel when I get up on a Monday morning.

I like to point out that a good team can also offer so much support at work. Your motivation and fulfillment can be influenced greatly by your colleagues.

3. Work-Life Balance

I like to think that we spend half of our waking hours working. That is, 8 out of 16 hours trading our skills for compensation.

You may work longer than 8 hours on some days due to circumstances. However, if overwork becomes consistent, something is not healthy with your current environment.

Realistically, not everyone is built to grind out 10+ hours everyday. This is not sustainable for yourself, and your team.

You also likely have aspirations outside of work, hobbies you want to pursue, and family members you want to spend time with.

Ensure that you have time for them.

Final Thoughts

This is just a general guide, based on my perspectives.

That said, I understand there can be inertia to “go”; It feels easier to stay, naturally. I have left out any discussions around evaluating the other opportunities available. This is purely focused on evaluating your current job.

My (check)list describes my definition of what work means to me. I would encourage you to think what work means for you and your family.

Happy to connect, and talk about this over coffee if you are in Tokyo!

#career #motivation #principles

buy Kelvin a cup of coffee ☕

How-to: encourage the read to click on links

Kelvin Tay — Sun, 16 Oct 2022 03:30:27 +0000

We communicate to our readers in documentations, responses, and also comments. Today's communication tools allows us to include links easily in many ways.

This sentence includes a link within itself, but does not stop you from reading the entire message.

I find such links useful when the linked article is supplementary to your message.

In situations where I want to direct the reader to click on the link explicitly, I find it best to simply show them raw below:https://www.w3.org/DesignIssues/LinkMyths.html

Personally, I find that showing the link raw (as-is) at the end helps:

draw attention, especially if the main point is about the linked article
to allow readers to inspect the link URL, which can help in credibility too

buy Kelvin a cup of coffee ☕

How to: Create Sequential Workflows in CircleCI pipeline

Kelvin Tay — Thu, 29 Sep 2022 14:03:03 +0000

Preface

In a CircleCI pipeline, workflows run independent of one another. As such, there is no built-in feature to ensure workflow B runs after workflow A.

However, you can still achieve ordering, through some trickery.

How

To achieve ordering, we simply set an approval job as the first job for workflow B.

# contrived snippet of a .circleci/config.yaml

workflows:
  aaa:
    jobs:
      - one
      - two
  bbb:
    jobs:
      - start:
          type: approval
      - next:
          requires:
            - start

Subsequent jobs in workflow B will only run when the approval job is approved. As such, you can “force” a wait, and only approve this job when workflow A is completed.

Note that this requires manual intervention, of course.

However, a benefit in this approach is that your team can take the time to confirm the outcomes of workflow A. For example, workflow A has deployed some infrastructure changes (e.g., terraform apply), and you prefer inspecting these changes before running workflow B.

One Step Further

You can automate this approval, at the end of workflow A, via the Approve a job API.

Specifically, you would need to create a job that does the following:

jobs:
  ...
  approve-workflow:
    parameters:
      workflow-name:
        type: string
        description: workflow name
      job-name:
        type: string
        description: name of approval job in workflow
    docker:
      - image: cimg/base:current
    steps:
      - run:
          name: Find Workflow ID for << parameters.workflow-name >>
          command: |
            curl -H "Circle-Token: $CIRCLE_TOKEN" https://circleci.com/api/v2/pipeline/<< pipeline.id >>/workflow > workflows.json
            WORKFLOW_ID=$(jq -r '.items | map(select(.name == "<< parameters.workflow-name >>")) | .[0].id' workflows.json)
            echo "export WORKFLOW_ID='${WORKFLOW_ID}'" >> $BASH_ENV
      - run:
          name: Find Job ID for << parameters.job-name >>
          command: |
            curl -H "Circle-Token: $CIRCLE_TOKEN" "https://circleci.com/api/v2/workflow/${WORKFLOW_ID}/job" > jobs.json
            APPROVAL_JOB_ID=$(jq -r '.items | map(select(.name == "<< parameters.job-name >>" and .type == "approval")) | .[0].id' jobs.json)
            echo "export APPROVAL_JOB_ID='${APPROVAL_JOB_ID}'" >> $BASH_ENV
      - run:
          name: Approve job
          command: |
            curl -X POST -H "Circle-Token: $CIRCLE_TOKEN" "https://circleci.com/api/v2/workflow/${WORKFLOW_ID}/approve/${APPROVAL_JOB_ID}" | jq .

In the spirit of sharing, I have created a CircleCI Orb that codifies the above job for your convenience.

https://circleci.com/developer/orbs/orb/kelvintaywl/control-flow

I hope this article and the Orb will be useful. Keep on building, folks!

#circleci #cicd #workflow

buy Kelvin a cup of coffee ☕