DEV Community: Ray Mathew

2024: Career Review

Ray Mathew — Mon, 30 Dec 2024 00:32:32 +0000

This year is different. I haven’t been employed for 10 months. But I’ve not been idle either.

Milestones and moments:

I left Thoughtworks
I had not wanted it to end though. I liked the culture, where developers were treated as the most important employees, and up-skilling was a high priority. I liked being in the presence of so many smart people. It made me a better person just by listening to them talk to each other.

But life rarely works the way you plan it. I moved to the US to be with my wife, and I had to resign. My biggest regret - not being able to work with more than one client in my entire 4 years there.
I do not have a mentor for the first time in my career.
A side effect of being unemployed. I had had always had one, even when I had become the Team Lead. Now, while I’m waiting for a new beginning in a new company, I don’t really have anyone. I guess it doesn’t matter that much, since I’m sure to find one again. But I think this thought matters to me because I felt sad when I had to say goodbye to my last mentor, who had left TW just before I did.
AWS Certification
Now that I had left the bubble of Thoughtworks and the tough client (who will continue to not be named), I needed to work on things that were in high demand in the industry. I decided to go for an AWS Developer Associate certification, since that was the buzzword I had heard most while at Thoughtworks, and also many people seemed to be getting jobs with that.

I spent 4-5 months studying for it. It was a lot to absorb. Even after 5 practice papers and maybe a few hundred questions, I couldn’t get more than 85%. There was so much to learn, and a lot of it could be acquired through real-world experience only.

But, I got it. And just in time - I also got my work permit approved in the same week.
The US dev job market is a nightmare.
I didn’t believe it when I read on Reddit how people applied to 500+ companies without getting an offer. Surely it can’t be that bad? I had cracked my very first application and interview every time I looked to switch in my career back in India!

But it was bad. I stopped keeping a track of the companies I had applied to - I was so fed up. Some companies responded with a rejection 4 months later, while others instantly rejected my resume within minutes.

I have applied to 300+ companies so far, out of which I got just 7 interviews. Referrals didn’t help either. Microsoft, Netflix, Airbnb, Amazon - all rejected my resume. I was sure that I would at least get an interview and dazzle them with my System Design skills.

My confidence wavered. I was second-guessing myself. For example,
- I decided to redo my resume in the boring format with Latex, in case it was getting auto-rejected by some tool.
- I created multiple resumes, for Frontend, Fullstack and TL roles.
- I was told by multiple people to emphasize my technical skills more, and my TL responsibilities less. What the hell! It was the opposite in India.
Out of the companies I did get interviews with - IBM, Nike, Amazon (on the 7th try) and Clipboard Health - I was rejected in the first round. And here I met my next roadblock. One I had been avoiding for 9 years.
DSA
I had tried to study it multiple times over the years, but always found it too difficult. Over time I decided that it was no longer important, because a good company would value the TL skills I have, and the huge tech stack I’ve worked with.

Nope. First off, there are very few TL roles available. And of the few, many jumped straight to Principal Engineer. And almost all the junior and senior levels roles required DSA knowledge. I didn’t even bother applying to the ones that said DSA was mandatory.

After 5 months, I decided I could no longer avoid it. I decided to take a paid course, practice some Leetcode, and get to applying to a lot more openings.

And that’s where I am right now. I’m doing a 5 course series on Algorithms over 4 months on Coursera. Let’s see where it gets me. On the plus side, I at least understand everything now, apart from the insane theorems in the CLRS textbook. I've already passed the first course with 99% grade.
I’ve decided to make a niche for myself.
This came through multiple motivations. First, I had a lot of free time now, and I didn’t want to spend all of it studying for interviews. Second, I still don't have much of an online reputation as a developer. Third, there are a lot of things to learn in the developer space. Fourth, I had been writing ideas for years on things I could build, but never found the time.

And so, the niche is this: Build a product that I or someone else would actually use. Make it like a real product that can be shipped. And build it using a language, framework or tool that’s famous but I've never used before. When I'm done, write about how I built it, and repeat the process with a new product.

In this way, I learn a ton of stuff, I finally get to work on my ideas, and I get to be known as ‘that guy who always builds stuff’. Maybe at some point I can even learn stuff like mechanical design, and combine it with coding.
What has come out of the niche so far.
1. An API written in Golang, to simulate Materia Fusion from Crisis Core.
  I am super proud of this. In the big picture, this is a very basic GET api with a ton of if-else conditions. But it’s my first ever project that took several weeks to complete. And it’s in a shippable state, ready to be used by others.
  In the process, I strengthened my Go knowledge and learnt how to use a Cloud service (Google), how to use PostgreSQL (a big mental block), and a Cloud DB service (CockroachDB).
2. A NextJS fullstack website, to showcase hikes and travels done by my wife and I.
  Another project I’m very proud of. Because my wife and I will actually use this. She is the primary consumer of this app.
  
  As of this writing, the desktop version is ready, and the mobile version is in the works. I don’t think I’ve done a great job with the UX. And I’m not sure how much I need to learn to get good at it.
  - Github link
  Through this I learnt NextJS (and about Vercel, the geniuses who made React fullstack) and, very importantly, Typescript. And I got to use Notion API as well, where all the data is stored.

Notable firsts:

Experienced the nightmare job market in US.
Learnt and used Cloud services.
Got a certification that actually matters in the industry.
Contributed significantly to open-source. https://github.com/Buzzpy/Dev-Encyclopedia (At the time of writing, I’m credited as the top contributor by the owner).

New things learnt:

AWS. I’ve only scratched the surface though.
Google Cloud.
CockroachDB.
Basics of PostgreSQL.
Typescript. Again, just the surface.
NextJS.
AstroJS.
Open Graph meta tags.
Entity tags.
A shit ton of DSA stuff.

Reflections:

Knowing what I know now after 6 months of job hunting, I would’ve studied DSA first before the AWS certification.
Taking this long a break from my career has made me truly appreciate that a job is a just a job, and money can be obtained through other sources too.
It’s hard to stay motivated for months when you have no concrete goals or deadlines. It requires constant discipline to keep building towards something.

My plan for 2025: Get a job. Work on side projects. Write. It’s about just getting my life in order. No grand ambitions for the first time.

Building an API with Go, PostgreSQL, Google Cloud and CockroachDB

Ray Mathew — Wed, 23 Oct 2024 19:27:29 +0000

I built an API with Go and PostgreSQL, set up a CI/CD pipeline with Google Cloud Run, Cloud Build, Secret Manager and Artifact Registry, and connected the Cloud Run instance to CockroachDB.

The API is based on the game Crisis Core: Final Fantasy VII, to simulate “Materia Fusion”. This article’s intended audience is for developers who just want to know how to build and deploy the API. I have another article where I talk about everything I learnt while working on this project, what didn’t work, and understanding and translating the game’s materia fusion rules (link coming soon).

Links for easy reference

API Objective

3 endpoints — health check (GET), list of all materia (GET), and simulate materia fusion (POST)

The Domain Model

Materia (both singular and plural) is a crystal orb that serves as a source of magic. There are 144 distinct materia in the game, and they’re broadly classified into 4 categories: “Magic”, “Command”, “Support” and “Independent”. However, for the purpose of figuring out the rules of materia fusion, it was easier to have 32 internal categories based on their fusion behaviour, and 8 grades within those categories (see reference).

A materia becomes ‘Mastered’ when it is used for a certain duration. The duration is not important here.

Most importantly, 2 materia can be fused to produce a new materia. The rules governing fusion are influenced by:

Whether either or both materia are mastered.
Which materia comes first (as in X+Y is not necessarily equal to Y+X).
Materia internal category.
Materia grade.

And there are a LOT of exceptions, with some rules having 3 levels of nested if-else logic. This eliminates the possibility of creating a simple table in the DB and persisting 1000+ rules into it, or coming up with One Formula To Rule Them All.

In short, we need:

A table materia with columns name(string), materia_type(ENUM) (the 32 internal categories), grade(integer), display_materia_type(ENUM) (the 4 categories used in the game), description(string) and id(integer) as an auto-incrementing primary key.
A data structure to encapsulate the basic rules format MateriaTypeA + MateriaTypeB = MateriaTypeC.
Code to use the basic and complex rules to determine the output Materia in terms of its internal category and grade.

1. Setting Up The Local PostgreSQL DB

Ideally you can install the DB from the website itself. But the pgAdmin tool could not connect to the DB for some reason, so I used Homebrew.

Installation

brew install postgresql@17

This will install a whole bunch of CLI binary files to help use the DB. Optional: add /opt/homebrew/opt/postgresql@17/bin to $PATH variable. Next, create the DB:

# create the DB
createdb materiafusiondb
# step into the DB to perform SQL commands
psql materiafusiondb

Create the user and permissions

-- create an SQL user to be used by the Go server
CREATE USER go_client WITH PASSWORD 'xxxxxxxx';

-- The Go server doesn't ever need to add data to the DB. 
-- So let's give it just read permission.
CREATE ROLE readonly_role;
GRANT USAGE ON SCHEMA public TO readonly_role;

-- This command gives SELECT access to all future created tables. 
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO readonly_role;

-- If you want to be more strict and give access only to tables that already exist, use this:
-- GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly_role;

GRANT readonly_role TO go_client;

Create the table

CREATE TYPE display_materia_type AS ENUM ('Magic', 'Command', 'Support', 'Independent');

CREATE TYPE materia_type AS ENUM ('Fire', 'Ice', 'Lightning', 'Restore', 'Full Cure', 'Status Defense', 'Defense', 'Absorb Magic', 'Status Magic', 'Fire & Status', 'Ice & Status', 'Lightning & Status', 'Gravity', 'Ultimate', 'Quick Attack', 'Quick Attack & Status', 'Blade Arts', 'Blade Arts & Status', 'Fire Blade', 'Ice Blade', 'Lightning Blade', 'Absorb Blade', 'Item', 'Punch', 'SP Turbo', 'HP Up', 'AP Up', 'ATK Up', 'VIT Up', 'MAG Up', 'SPR Up', 'Dash', 'Dualcast', 'DMW', 'Libra', 'MP Up', 'Anything');

CREATE TABLE materia (
    id integer NOT NULL,
    name character varying(50) NOT NULL,
    materia_type materia_type NOT NULL,
    grade integer NOT NULL,
    display_materia_type display_materia_type,
    description text
    CONSTRAINT materia_pkey PRIMARY KEY (id)
);

-- The primary key 'id' should auto-increment by 1 for every row entry.
CREATE SEQUENCE materia_id_seq
    AS integer
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;

ALTER SEQUENCE materia_id_seq OWNED BY materia.id;

ALTER TABLE ONLY materia ALTER COLUMN id SET DEFAULT nextval('materia_id_seq'::REGCLASS);

Add the data

Create an Excel sheet with table header and data, and export it as a CSV file. Then run the command:

COPY materia(name,materia_type,grade,display_materia_type,description) FROM
 '<path_to_csv_file>/materiadata.csv' DELIMITER ',' CSV HEADER;

2. Creating the Go Server

Create the boilerplate code using autostrada.dev. Add the options of api, postgresql, httprouter , env var config, tinted logging, git, live reload, makefile. We end up getting a file structure like this:

📦 codebase
├─ cmd
│  └─ api
│     ├─ errors.go
│     ├─ handlers.go
│     ├─ helpers.go
│     ├─ main.go
│     ├─ middleware.go
│     └─ server.go
├─ internal
│  ├─ database --- db.go
│  ├─ env --- env.go
│  ├─ request --- json.go
│  ├─ response --- json.go
│  └─ validator
│     ├─ helpers.go
│     └─ validators.go
├─ go.mod
├─ LICENSE
├─ Makefile
├─ README.md
└─ README.html

.env file

The boilerplate generator has created code to fetch environment variables and add them to the code, but we can make it easier to track and update the values.

Create <rootfolder>/.env file. Add the following values:

HTTP_PORT=4444
DB_DSN=go_client:<password>@localhost:5432/materiafusiondb?sslmode=disable
API_TIMEOUT_SECONDS=5
API_CALLS_ALLOWED_PER_SECOND=1

Add the godotenv library:

go get github.com/joho/godotenv

Add the following to main.go:

// At the beginning of main():
err := godotenv.Load(".env") // Loads environment variables from .env file
if err != nil { // This will be true in prod, but that's fine.
  fmt.Println("Error loading .env file")
}


// Modify config struct:
type config struct {
  baseURL string
  db      struct {
    dsn string
  }
  httpPort                 int
  apiTimeout               int
  apiCallsAllowedPerSecond float64
}

// Modify run() to use the new values from .env:
cfg.httpPort = env.GetInt("HTTP_PORT")
cfg.db.dsn = env.GetString("DB_DSN")
cfg.apiTimeout = env.GetInt("API_TIMEOUT_SECONDS")
cfg.apiCallsAllowedPerSecond = float64(env.GetInt("API_CALLS_ALLOWED_PER_SECOND"))

// cfg.baseURL = env.GetString("BASE_URL") - not required

Middleware and Routes

The boilerplate already has a middleware to recover from panics. We will add 3 more: Content-Type checking, rate-limiting and API timeout protection.

Add tollbooth library for rate-limting:

go get github.com/didip/tollbooth

Update <rootfolder/api/middleware.go:

func (app *application) contentTypeCheck(next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  if r.Header.Get("Content-Type") != "application/json" {
   app.unsupportedMediaType(w, r)

   return
  }
  next.ServeHTTP(w, r)
 })
}


func (app *application) rateLimiter(next http.Handler) http.Handler {
 limiter := tollbooth.NewLimiter(app.config.apiCallsAllowedPerSecond, nil)
 limiter.SetIPLookups([]string{"X-Real-IP", "X-Forwarded-For", "RemoteAddr"})

 return tollbooth.LimitHandler(limiter, next)
}


func (app *application) apiTimeout(next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  timeoutDuration := time.Duration(app.config.apiTimeout) * time.Second

  ctx, cancel := context.WithTimeout(r.Context(), timeoutDuration)
  defer cancel()

  r = r.WithContext(ctx)

  done := make(chan struct{})

  go func() {
   next.ServeHTTP(w, r)
   close(done)
  }()

  select {
  case <-done:
   return
  case <-ctx.Done():
   app.gatewayTimeout(w, r)
   return
  }
 })
}

The middleware need to be added to the routes. They can be either added to all the routes, or to specific ones. In our case, Content-Type checking (that is, mandating the input headers to include Content-Type: application/json) is only needed for POST requests. So modify routes.go as follows:

func (app *application) routes() http.Handler {
 mux := httprouter.New()

 mux.NotFound = http.HandlerFunc(app.notFound)
 mux.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowed)

 // Serve the Swagger UI. Uncomment this line later
 // mux.Handler("GET", "/docs/*any", httpSwagger.WrapHandler)

 mux.HandlerFunc("GET", "/status", app.status)
 mux.HandlerFunc("GET", "/materia", app.getAllMateria)

 // Adding content-type check middleware to only the POST method
 mux.Handler("POST", "/fusion", app.contentTypeCheck(http.HandlerFunc(app.fuseMateria)))

 return app.chainMiddlewares(mux)
}

func (app *application) chainMiddlewares(next http.Handler) http.Handler {
 middlewares := []func(http.Handler) http.Handler{
  app.recoverPanic,
  app.apiTimeout,
  app.rateLimiter,
 }

 for _, middleware := range middlewares {
  next = middleware(next)
 }

 return next
}

Error handling

Add the following methods to <rootfolder>/api/errors.go to help the middleware functions:

func (app *application) unsupportedMediaType(w http.ResponseWriter, r *http.Request) {
 message := fmt.Sprintf("The %s Content-Type is not supported", r.Header.Get("Content-Type"))
 app.errorMessage(w, r, http.StatusUnsupportedMediaType, message, nil)
}

func (app *application) gatewayTimeout(w http.ResponseWriter, r *http.Request) {
 message := "Request timed out"
 app.errorMessage(w, r, http.StatusGatewayTimeout, message, nil)
}

Request and Response structure files

<rootfolder>/api/dtos.go for the responses of all 3 endpoints:

package main

// MateriaDTO provides Materia details - Name, Description and Type (Magic / Command / Support / Independent)
type MateriaDTO struct {
 Name        string `json:"name" example:"Thunder"`
 Type        string `json:"type" example:"Magic"`
 Description string `json:"description" example:"Shoots lightning forward dealing thunder damage."`
}

// StatusDTO provides status of the server
type StatusDTO struct {
 Status string `json:"Status" example:"OK"`
}

// ErrorResponseDTO provides Error message
type ErrorResponseDTO struct {
 Error string `json:"Error" example:"The server encountered a problem and could not process your request"`
}

<rootfolder>/api/requests.go for the POST \fusion endpoint:

package main

import (
 "github.com/RayMathew/crisis-core-materia-fusion-api/internal/validator"
)

// MateriaFusionRequest provides input Materia names and their Mastered states
type MateriaFusionRequest struct {
 Materia1Mastered *bool               `json:"materia1mastered" example:"true"`
 Materia2Mastered *bool               `json:"materia2mastered" example:"false"`
 Materia1Name     string              `json:"materia1name" example:"Fire"`
 Materia2Name     string              `json:"materia2name" example:"Blizzard"`
 Validator        validator.Validator `json:"-"`
}

Validator, from the generated code, will be used later to validate the input fields for \fusion endpoint.

Data Structure for the combination rules

Create the file <rootfolder>/internal/crisis-core-materia-fusion/constants.go

Add the following:

package crisiscoremateriafusion

type MateriaType string

const (
 Fire              MateriaType = "Fire"
 Ice               MateriaType = "Ice"
 Lightning         MateriaType = "Lightning"
 Restore           MateriaType = "Restore"
 ...

full list of 32 MateriaTypes can be found here.

Create the file <rootfolder>/internal/crisis-core-materia-fusion/models.go

Add the following:

package crisiscoremateriafusion

type Materia struct {
 Name        string `json:"name"`
 Type        string `json:"materia_type"`
 DisplayType string `json:"display_type"`
 Description string `json:"description"`
 Grade       int    `json:"grade"`
}

// Data structure to hold all basic combination rules
type BasicCombinationRule struct {
 FirstMateriaType     MateriaType
 SecondMateriaType    MateriaType
 ResultantMateriaType MateriaType
}

var FILBasicRules = []BasicCombinationRule{
 {Fire, Fire, Fire},
 {Ice, Ice, Ice},
 {Lightning, Lightning, Lightning},
 {Fire, Ice, Lightning},
 {Ice, Fire, Lightning},
 {Fire, Lightning, Ice},
 {Lightning, Fire, Ice},
 {Lightning, Ice, Fire},
 {Ice, Lightning, Fire},
}

var ...

full list of rules can be found here.

Handler for GET `\materia` in `api/handlers.go`

func (app *application) getAllMateria(w http.ResponseWriter, r *http.Request) {
 var allDisplayMateria []MateriaDTO
 var allMateria []ccmf.Materia
 var err error

 allMateria, err = app.getAllMateriaFromApprSource()

 if err != nil {
  app.serverError(w, r, err)
 }

// Some materia have the same name but different grades. 
// We need to allow only unique names are sent in the response.
 seenMateriaNames := make(map[string]bool)
 for _, materia := range allMateria {
  if _, isDuplicate := seenMateriaNames[materia.Name]; !isDuplicate {
   seenMateriaNames[materia.Name] = true
   allDisplayMateria = append(allDisplayMateria, MateriaDTO{
    Name:        materia.Name,
    Type:        materia.DisplayType,
    Description: materia.Description,
   })
  }
 }

 err = response.JSON(w, http.StatusOK, allDisplayMateria)
 if err != nil {
  app.serverError(w, r, err)
 }
}

func (app *application) getAllMateriaFromApprSource() (allMateria []ccmf.Materia, err error) {
 // Check if allMateria data is in cache
 if data, found := app.getCachedData(string(ccmf.AllMateriaCacheKey)); found {
  // Type assertion: assert that data is of type []Materia
  if allMateriaCache, ok := data.([]ccmf.Materia); ok {
   allMateria = allMateriaCache
   app.logger.Debug("cache hit")
  } else {
   app.logger.Error("Failed to assert cached data as []Materia")
   return nil, errors.New("failed to assert cached data as []Materia")
  }
 } else {
  // allMateria data is not in cache. Get from DB
  app.logger.Debug("cache miss")
  allMateria, err = app.db.GetAllMateria()
  app.setCache(string(ccmf.AllMateriaCacheKey), allMateria)
 }
 return
}

In-server cache

We are using an in-server cache because:

The data fetched from the DB never changes.
The same data is used by both \materia and \fusion endpoints.

Update main.go:

// declare a cache and a mutex.
// the mutex is to ensure there is only one operation using the cache at a time.
type application struct {
 db     *database.DB
 logger *slog.Logger
 cache  map[string]interface{}
 wg     sync.WaitGroup
 mu     sync.Mutex
 config config
}

// in run() initialize the cache:
app := &application{
  config: cfg,
  db:     db,
  logger: logger,
  cache:  make(map[string]interface{}),
 }

Update api/helpers.go:

// remove backgroundTask()
// add getter and setter for the cache:

func (app *application) getCachedData(key string) (interface{}, bool) {
 app.mu.Lock()
 defer app.mu.Unlock()

 data, found := app.cache[key]
 return data, found
}

func (app *application) setCache(key string, value interface{}) {
 app.mu.Lock()
 defer app.mu.Unlock()

 app.cache[key] = value
}

Handler for POST `\fusion` in `api/handlers.go`

// showing only relevant parts of the code

func (app *application) fuseMateria(w http.ResponseWriter, r *http.Request) {
 var fusionReq MateriaFusionRequest
 err := request.DecodeJSON(w, r, &fusionReq)
 if err != nil {
  app.badRequest(w, r, err)
  return
 }

// Using the Validator we had defined in dtos.go
 fusionReq.Validator.CheckField(fusionReq.Materia1Name != "", "materia1name", "materia1name is required")
 fusionReq.Validator.CheckField(fusionReq.Materia2Name != "", "materia2name", "materia2name is required")
 fusionReq.Validator.CheckField(fusionReq.Materia1Mastered != nil, "materia1mastered", "materia1mastered is required")
 fusionReq.Validator.CheckField(fusionReq.Materia2Mastered != nil, "materia2mastered", "materia2mastered is required")

 if fusionReq.Validator.HasErrors() {
  app.failedValidation(w, r, fusionReq.Validator)
  return
 }

 var allMateria []ccmf.Materia

 allMateria, err = app.getAllMateriaFromApprSource()

 if err != nil {
  app.serverError(w, r, err)
 }

 var materia1Type string
 var materia1Grade int
 var materia2Type string
 var materia2Grade int

// matching the request input with the categories in DB
 for _, materia := range allMateria {
  if materia1Type != "" && materia2Type != "" {
   break
  }
  if materia.Name == fusionReq.Materia1Name && materia1Type == "" {
   materia1Type = materia.Type
   materia1Grade = materia.Grade
  }
  if materia.Name == fusionReq.Materia2Name && materia2Type == "" {
   materia2Type = materia.Type
   materia2Grade = materia.Grade
  }
 }

 if materia1Type == "" || materia2Type == "" {
  app.badRequest(w, r, errors.New("one or both of the Materia names are not recognised"))
  return
 }

// game rule - higher grade Materia moves to first position
 exchangePositionsIfNeeded(&fusionReq, &materia1Grade, &materia2Grade, &materia1Type, &materia2Type)

 relevantBasicRuleMap := ccmf.BasicRuleMap[ccmf.MateriaType(materia1Type)]
 var relevantBasicRule ccmf.BasicCombinationRule

// finding the relevant combination rule
 for _, rule := range relevantBasicRuleMap {
  if (rule.FirstMateriaType == ccmf.MateriaType(materia1Type)) &&
   (rule.SecondMateriaType == ccmf.MateriaType(materia2Type)) {
   relevantBasicRule = rule
   break
  }
 }

 var resultantMateria MateriaDTO

// game rule - grade of resultant Materia depends on the input Materia as well as their Mastered state
 resultantMateriaGrade := determineGrade(fusionReq, materia1Grade)

 if relevantBasicRule.FirstMateriaType == "" {
  app.logger.Info("none of the basic rules satisfy the requirement.")

  // get final output using complex rules
  resultantMateria = useComplexRules(materia1Grade, materia2Grade, resultantMateriaGrade, materia1Type, materia2Type, *fusionReq.Materia1Mastered, *fusionReq.Materia2Mastered, &allMateria)
 } else {
  // get final output using basic rules
  resultantMateriaType := relevantBasicRule.ResultantMateriaType

  for _, materia := range allMateria {
   if materia.Grade == resultantMateriaGrade && materia.Type == string(resultantMateriaType) {
    resultantMateria.Name = materia.Name
    resultantMateria.Type = materia.DisplayType
    resultantMateria.Description = materia.Description
    break
   }
  }
 }


 err = response.JSON(w, http.StatusOK, resultantMateria)
 if err != nil {
  app.serverError(w, r, err)
 }
}

// Combination rules which do not follow any pattern, and had to be coded separately
func useComplexRules(materia1Grade, materia2Grade, resultantMateriaGrade int, materia1Type, materia2Type string, materia1Mastered, materia2Mastered bool, allMateria *[]ccmf.Materia) (resultantMateria MateriaDTO) {
 var resultantMateriaType string

 switch {
 // Complex Rule 1: FIL, Defense
 case (materia1Type == string(ccmf.Fire) ||
  materia1Type == string(ccmf.Ice) ||
  materia1Type == string(ccmf.Lightning)) && materia2Type == string(ccmf.Defense):
  if materia1Grade == 1 && materia2Grade == 1 {
   // output is Defense when grades are equal to 1
   resultantMateriaType = string(ccmf.Defense)
   if materia1Mastered || materia2Mastered {
    // final Grade is increased when output is Defense
    increaseGrade(&resultantMateriaGrade)
   }
  } else {
   // output is FIL when grades are NOT equal to 1
   resultantMateriaType = materia1Type
  }

  ... 

// prepare response DTO
 updateResultantMateriaData(allMateria, resultantMateriaGrade, resultantMateriaType, &resultantMateria)
 return resultantMateria
}

Complete handler code can be found here.

Swagger UI and OpenAPI definition doc

Add the Swagger library:

go get -u github.com/swaggo/swag/cmd/swag
go get github.com/swaggo/http-swagger
go get github.com/swaggo/swag

In routes.go uncomment the Swagger line, and add the import:

httpSwagger "github.com/swaggo/http-swagger"

In the handler, DTO and model files, add comments for Swagger documentation. Refer this for all options.

In the terminal, run:

cd api
swag init -d .

This creates an api/docs folder, with the definition available for Go, JSON and YAML.

To test it, start the local server and open http://localhost:4444/docs.

Final folder structure:

📦 crisis-core-materia-fusion-api
.gitignore
├─ Dockerfile
├─ LICENSE
├─ Makefile
├─ README.md
├─ api
│  ├─ docs
│  │  ├─ docs.go
│  │  ├─ swagger.json
│  │  └─ swagger.yaml
│  ├─ dtos.go
│  ├─ errors.go
│  ├─ handlers.go
│  ├─ helpers.go
│  ├─ main.go
│  ├─ middleware.go
│  ├─ requests.go
│  ├─ routes.go
│  └─ server.go
├─ go.mod
├─ go.sum
└─ internal
   ├─ crisis-core-materia-fusion
   │  ├─ constants.go
   │  └─ models.go
   ├─ database -- db.go
   ├─ env -- env.go
   ├─ request -- json.go
   ├─ response -- json.go
   └─ validator
      ├─ helpers.go
      └─ validator.go

3. Setting up the remote PostgreSQL instance in CockroachDB

Use the steps from here.
After creating the certificate, create <rootfolder>/certs/root.crt in the project and add the certificate there. We will make a reference to this file later in the Google Run configuration.
CAUTION! We are NOT pushing this folder to the remote repository. Add certs/ folder to .gitignore. We are creating the certificate in local only to test the connection, if you wish.
Now when you go to CockroachDB → Dashboard → Left Menu → Databases, you should be able to see the DB you created.

Data Migration

From your local DB instance, run:

pg_dump --no-owner --no-privileges -U <admin_username> -d materiafusiondb > full_dump.sql

Go to CockroachDB → Left Menu → Migrations → Add Schema → Drag the SQL file you just got. All the steps will run, except the table data insertion. It will also show you a list of steps that were executed.
At the time of writing this article, the PostgreSQL instance in CockroachDB doesn’t support statements like IMPORT INTO. So I had to create an INSERT statement in a local SQL file for 270 rows (which we can derive from the pg_dump output we just got).
Log into the remote instance, and run the SQL file.

Logging in to the remote instance:

psql -h <REMOTE_DB_CLUSTER_HOSTNAME> -U <REMOTE_USERNAME> -d materiafusiondb -p <REMOTE_DB_PORT>

4. Deploy a Google Cloud Run instance

Create a Dockerfile like this.
Go to Google Cloud Run and create a new project for the API.
Create Service → Continuously deploy from a repo → SETUP WITH CLOUD BUILD → Repository Provider = Github → Select your repo → Build Type = Dockerfile → Save.
Authentication = Allow unathenitcated invocations.
Most of the defaults should be fine as is.
Scroll down to Containers → Container Port = 4444.
Select Variables and Secrets tab, and add the same environment variables as we have in our local .env file.

Values:

HTTP_PORT = 4444
DB_DSN = <remote_cockroachdb_url>?sslmode=verify-full&sslrootcert=/app/certs/root.crt
API_TIMEOUT_SECONDS = 5
API_CALLS_ALLOWED_PER_SECOND = 1

Using Google Secret Manager for the certificate

The last piece of the puzzle.

Search for Secret Manager → Create Secret → Name = ‘DB_CERT’ → Upload the .crt certificate of the CockroachDB.
In Cloud Run → (your service) → Click Edit Continuous Deployment → Scroll down to Configuration → Open Editor.
Add this as the first step:

  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
    args:
      - '/bin/sh'
      - '-c'
      - >
        mkdir -p certs

        gcloud secrets versions access latest --secret="DB_CERT" >
        certs/root.crt
    id: Fetch Secret

This will make Cloud Build create the file certs/root.crt in our project before the build starts, so that the Dockerfile will have access to it even though we never pushed it to our Github repository.

And that’s it. Try pushing a commit and check if the build triggers. The Cloud Run dashboard will show the URL of your hosted Go server.

For questions related to “Why did you do X and not Y?” read this.

For anything else that you want to know or discuss, go here, or comment below.

2023: A career review

Ray Mathew — Tue, 14 May 2024 17:07:18 +0000

Milestones and moments:

Diversified into Data Mesh.
Just as I had planned last year, I got into the theory of Data Mesh. It’s useful info for the future, but not something I’m able to utilize right now. I had no idea that the person who coined the term was a Thoughtworker! (Zhamak Dehghani)
Delays in TL role approval.
I had already been feeling hesitant about trying for the TL with the recent management changes in the client company. I think my own TL had convinced me to try once more, given that I had already put in so much work. So I did. But things didn't really move along for at least 5 months.

I was interviewed multiple times on the Thoughtworks side - the questions were fairly easy, mainly about what I would do in various technical and business related scenarios.

I was also given me a formal feedback from the client, that I was the most capable in the team of taking up the TL role, but I needed to stop relying on my own TL to get me out of difficult spots. Admittedly, I felt exposed. I DID in fact do that with both my previous TLs.
Real world Kafka and Kubernetes experience.
Small achievement in the big scale of things, but a big mental breakthrough for me. I had been intimidated of using big systems like Kafka and Kubernetes, because for so long they were just names brought up frequently, signifying their importance and abstraction of the systems we were working on. Not to mention they took me a while to learn. It was not the same concept as learning a language or React.
But I can now say that I’ve worked on both Kafka and Kubernetes.
Unanimous support for TL role.
All my teammates said I was ready to lead the team. My current TL said I was ready. My mentor, also a TL, said I was ready.

So why was there a delay? I'm not sure...
Finally became TL. Crap, now what?
It finally happened. I had basically given up the thought of being TL and was looking to move on to another account. They had even gotten someone who was apparently supposed to be the new TL.

But then it happened. I was told I was going to be the next TL during an in-person training session. I panicked. I had the responsibility of the team’s success or failure now. No one to fall back on. I should’ve maybe been celebrating more? It was a massive moment for my career.
Absolutely overloaded in the first few months.
It’s the most I’ve ever worked in my life. I worked 9 continuous days (including Saturday and Sunday), before I felt a bit sick and had to stop. I was waking up at 6am, and working till 11pm. Getting requirements ready for the team, hashing out solutions with the other teams, creating contracts, etc. I experienced for the first time what it was like where I was the bottleneck of a team’s output, and everybody wanted my time. From the stakeholders to the Devs to the BA. People called me on the phone with no hesitation, assuming that I’m always available to answer their “one minute” question which would turn into 20 minutes. Devs would either be waiting for me to tell them what to do, or skipping code reviews because I looked busy.

We got in serious trouble when, by just the 3rd Sprint, we had 16 points of spillover, and the client's upper management were asking what was wrong.

For the first time in my life, I became that guy who ignored calls. I had to. I also ignored unplanned Zoom meetings. I couldn’t be the “helpful guy” to everyone anymore, except my own team. They were the first priority. I decided that the BA and IM alone would go to meetings to clarify requirements, and I would join when things got technical. After all, that should’ve been the extent of my role. But Thoughtworks always expected the TLs to be Jacks of all trades.

Things got maybe 5% percent less stressful by taking the above measures. But I was still working a hell lot, I hardly exercised, I lost weight, and I must’ve worked at least 9 Sundays in 6 months.
Got to implement my strengths - not tech, but people skills.
1) I managed to make the team meet the clients upper management. I believed it was very important, for the team to feel a sense of ownership, to put a face on the people who were telling them what to do for 10 hours a day, 50 weeks in a year. I was the only one who even gave it any importance.
2) I made a huge bit of progress in motivating one my team members to be more proactive and take up solo tasks.
3) I convinced one of my mentee’s TL to not give her a “Needs Improvement” rating, as it would demotivate her. It took a full 30 minutes. Some of the most difficult talking and negotiation I’ve ever done.]
The delegation problem.
This was my biggest failure as a team lead. To delegate, and to make the team more autonomous.
The reason why I was such a bottleneck for the team was that they were always waiting on me to tell them what to do next. And when they did do something, they would miss some of the impacts it had on other parts of code. How does a team lead know the answer to this question - was the bottleneck failure due to him / her not empowering the team more and preparing more clear instructions in advance? Or was it because the team was not mature / interested enough to take ownership, and initiative? Is being in charge of hiring the only solution?

I don’t yet have an answer to any of these questions today.
Some difficulties and learnings.
I’ll try to keep this brief.

1) Every requirement was urgent. So, nothing was. We had to make our own assessment of what needed to be delivered first.

2) Unreasonable deadlines and expectations. For example, one of the folks I reported to expected me to write the pseudocode, error codes, error msgs, FSM diagram etc for the entire user journey in a week, so that the team had a template to follow. The actual feature ended up taking 6 months and 8 developers.

3) Office politics. For example, there was a guy who frequently lied to get his way, and his superiors turned a blind eye. There was a stakeholder who didn't understand tech, but came into meetings claiming that I should've caught some bug before it happened. There was a team who claimed that we would do "consumer driven development", so that my team, the consumers, had the responsibility of creating contracts. But when we wanted to make critical changes later on, they refused, claiming that they cannot alter their internal systems for our benefit.

Managing all of these and more required a lot of discussions, meetings, and heated arguments. It was not at all what I expected when I was starting my career as a developer.
In the end, it was a success.
Despite all these headwinds, we did it. We went into CUG testing early December, with very few bugs, most of which were a data issue.

We. Did. It. The nightmare that I endured to my mind and body was worth it. I could tell people for the rest of my life and my career, that I was TL for the first time in the most high profile and difficult TW project, and we delivered a product that the client was happy with. I got praise for the amount of workload and coordination I took on myself. I got praise from my team, including the very touching “Best TL I’ve worked with”.
Zero interviews.
Nothing to add here. I just didn’t have the time to take out 2 hours of my day to interview someone when I was expected to deliver a feature before the requirements were ready.

Notable firsts:

I became TL.
I helped take a feature live as TL.
Diversified into Data Mesh, got practical experience in Kafka and Kubernetes.

New things learnt:

Practical implementation of Finite State Machine in Backend for a user journey.
Data Mesh.
Managing clients.
Managing a Dev and QA team.
Golang (strengthening concepts).

A very small list this time.

Reflections:

Being a TL for the first time with TW India’s most difficult client, with their most high profile feature, was harrowing. I lost sleep, weight, and any semblance of a personal life and leisure activities. I felt the urge to quit within just a month. But I persevered, and thankfully it was worth it. For the rest of my life I’ll be able to carry the confidence that if I could do something that difficult, I’m a lot more capable than I give myself credit for. Not to mention, we took the feature live, with no major bugs in CUG or Prod.
I’m glad I was able secure my exit after just 7 months in that project. It was terrible for my health, and no one should ever have to work like that.
The moment you climb the ladder and start managing people, office politics and mind games become visible. It’s something everyone unfortunately needs to learn to handle, so that they’re not taken advantage of.
It may not have occupied a lot of my time or effort, but being firm with teammates who you used to be your peers and friends is one of the most difficult things to do.

My plan for 2024: Relax. I’m going on a government imposed sabbatical, since I need a work permit once I enter US on an H4 visa. Also, learn things outside of my client's requirements for the first time in a while, and then look for a new job.

2022: A career review

Ray Mathew — Wed, 21 Dec 2022 11:08:33 +0000

Milestones and moments:

Completed my temp co-Tech Lead reign.
2 weeks of being a temp co-TL along with a teammate ended on 7th Jan. I was ashamed at the time to admit to myself that he had done a better job than I had. He could’ve probably even managed on his own. I kept zoning out, unable to track the 20+ problems or provide good solutions. My speaking skills managed to salvage some points.
I also didn’t like the experience I just had as co-TL. It was too much responsibility; I couldn’t take a break for a minute. What a reality check it was to my ambitions. But I didn’t want to tell this to my mentor and disappoint him.
Moved to a new sub-squad to assist the new TL.
A 3rd sub-squad in 2 years, with a new TL. I was initially in the running to be the TL myself, but the management team (rightly, in my opinion) hired someone new to take over. And I was to help him to settle in to the Thoughtworks (TW) and the project's ways of working. My mentor said that this was a good opportunity for me to have increased levels of responsibilities, and I had to agree he was right. But it also meant I would no longer be working with him. It was the end of a great period for me, and a move on to something more challenging.
I did not like my new role in the initial few weeks. I was spending more time in meetings with the TL and the clients than writing any code. I felt like a fraud. I was earning a chunk of money to just talk and listen.
I was also sometimes being called upon instead of the new TL to help deal with problems that required a historical context. I was too busy sulking at the time to realise that this was a great boost to my reputation as a lead.
A heavy dose of architecture.
I watched over 140 Developer to Architect videos over the year, and had subscribed to newsletters to keep up with the latest tools that an architect would need. This was the second major step (after becoming the stand-in TL) towards something bigger than writing code.
Gave an internal and an external talk.
I noticed my nervousness was much less than it had been in 2020. Another one of my goals (speaking) was progressing well.
I am finally part of a team that took a feature live.
It’s been 8 years, and I had previously always left a team just before the feature / product was released. And this time I just so happened to be the temp TL when it happened. A great boost to my confidence and my credibility as a future leader.
I helped take a microsite to production.
My biggest achievement this year. My TL encouraged me to lead the efforts in getting a microsite built and deployed all the way till production. This involved co-ordinating with multiple teams, keeping the clients updated, clarifying issues for other developers, and debugging in prod. Basically, being a team lead for a feature.
Took sole charge of debugging prod issues after the initial month of release.
Again, a bit of luck. The TL was not available. I was reading the prod logs alone, and reporting on them. This led to the client reaching out to me directly for subsequent issues.
The team has started to consider me as a lead.
I had been acting as a substitute TL for many months now whenever the TL was not around. But the day I started explaining our Sprint plans to the team was the day they finally accepted that I too was a capable lead in his absence. I hadn’t been looking for their approval, but it boosted my confidence none the less that I was doing something right.
Went to office for the first time since the pandemic started on 26th Aug, 2022.
I now find the role of a mentor easy.
As of the end of the year, I’m working with 9 mentees (one of the highest in TW India) and have gotten good reviews from them.
Interviews on hold.
I conducted very few interviews this year, due to the pressure of our delivery.

Notable firsts:

Was part of a team that took a product live.
Led the efforts in getting a feature built and deployed to prod.
Acted as temp TL consistently throughout the year.
Involved in the planning stages of the team’s work.

New things learnt:

nmap (basics)
BDD testing, using ginkgo and gomega
Distributed logging, via streaming and subscribing
Fitness functions
Observability
Kubernetes (basics)
Monorepos
GoCD configuration
Webpack configuration
Web 3.0
React Native Deeplinking (basics)
https://json-schema.org/
Git worktrees, git squash.
CryptoJS.
Websockets.

Reflections:

There were 3 major themes this year: learning architecture, getting face-time and trust with clients, and leading the efforts in building a feature.
I have completely conquered one of my weaknesses from last year - context switching. It was achieved by sheer brute force practice.
I had disliked the taste of the TL role at the start of the year - the messiness and the chaos. I just wanted to write code. But by the end of the year I realised that building systems is inherently a messy process and it’s a marvel anything works at all. Only a conscientious person can get the job done. I am now okay with what the role requires.
No matter how much we talk about values and culture, money matters most. And one must be wary of moonshot projects during recession.

My plan for 2023: Non-tech skills of a leader, and Data. It’s time to diversify my skillset.

2021: A career review

Ray Mathew — Sat, 29 Jan 2022 07:25:23 +0000

Continuing the tradition I started last year (29 days late).

Milestones and noteworthy moments:

Finally started giving showcases to clients.
Despite being in a project which follows Agile Methodology for a year I had not yet had the opportunity. I have now done it several times, and been almost always appreciated. And yet, the stage fright makes me reluctant on the next occasion.
Gave 4 talks in 4 months.
I improved on the streak from the previous year, and made myself even more comfortable with giving talks than ever before. But I still have a long way to go before I feel truly at ease on stage (virtual or otherwise). By the time I had given 3 talks I had wanted to take the rest of the year off from the palpitating pressure of doing any more.
Became obsessed with work in the beginning of the year.
I worked in 3 teams simultaneously for the first 3 months. I don’t know if it was because I wanted to ride the high of recent good reviews, or because I was using work as a welcome distraction from dealing with personal problems. Probably both. I worked for at least 11 hours on weekdays and prepped for the next week on weekends. There wasn't a moment for aimless thoughts.
Had a performance review that was akin to a therapy session.
The review I had in February with the Office Principal(OP), People Champion(PC) and my mentors will probably never happen again, since Thoughtworks is expanding rapidly and formalising procedures. It involved me talking freely about my journey over the last 1.2 years, my mentors saying nice things about me, and concluding with the OP and PC giving advice on how to carry on in the future. There was no judgement, or even a business-like agenda to the meeting. I left the meeting feeling lighter and happier.
Started conducting interviews.
Being an interviewer is a position of privilege and power. It has to be treated very carefully, as you hold another person’s career in your hands in that moment. It’s also about trust from the company, as your decision could make the difference between a mediocre and a star employee being hired that could have ripple effects years into the future.
Listening to the candidates talk about their work and projects also expanded my own knowledge. I loved doing it, despite it taking a lot of energy to listen carefully and ask calibrated questions for 1.5 hours.
Became a mentor and 'catalyst' to several people.
Again, a privilege and an honour. To put things in perspective, there was a time when I used to introduce myself to senior developers with "I'm sorry, I don't have a Computer Science background. Could you tell me what to do?". And now I give advice like I actually have something worthwhile to say. Surreal.
I have wondered many times though, if I had done enough to help my mentees, or if I could have done more. They themselves might not know the full answer to that question. How can one tell? How does one distinguish between a person who wants to try things on their own versus a person who does not want to be a burden by asking for more help?
Became obsessed with improving efficiency of my team in the middle of the year.
The '1% improvement everyday' idea had taken a hold on me. I was constantly looking for the smallest adjustments not just in our codebase, but in our team as well. For example:
- Learnings were to be shared with everyone so that no one is reinventing the wheel.
- Speed feedback sessions to find everyone's blind spots.
- Creating docs for standard procedures so that no one wastes time asking how to do something twice.
- Command aliases that automated several steps in our daily work.
Acted as a shadow trainer.
Yet some more luck, where I was named a shadow trainer for a workshop because all the student slots had already been taken up. This lead to opportunities where I could’ve become a permanent trainer as well.
Wrote an article after ages
My Ambition Has Evolved Beyond Being a Technical Expert
Learnt a little Golang and made my first steps towards being a full stack dev.
Worked on a couple of Stories, some refactoring, and fixed bugs. This has been a very important step towards me becoming a full stack developer, and making me more eligible to be a Tech Lead in the future. My own Tech Lead has been incredibly patient - he walked me through the solutions himself instead of pairing me off with another developer.
IPO.
I was in the right place at the right time with the Thoughtworks IPO. Being part of an IPO was a bucket list item, and I had never expected it to be fulfilled so soon.
Temporary co-Tech Lead.
A teammate and I were made temporary co-TLs of our team for 3 weeks. The word 'privilege' is starting to lose meaning by now. It was a massive learning curve, and one I need to work more on - particularly context switching.

Notable firsts:

Worked as a judge in a competition (Technovation Girls).
Became a mentor.
Gave 4 talks in a year.
Conducted interviews, and learnt to anchor them.
Worked on backend tasks again for the first time in 6 years.
Named temporary co-Tech Lead.

New things learnt:

Add git hooks to a project, using husky.
Configuring prettier and eslint config files.
Dealing with env variables and processes using NodeJS.
Learnt how to make the console pretty with NodeJS.
git stash (in depth), recursive modules and general commands in depth (link)
Docker (re-learnt it after forgetting much of what I learnt last year). Made extensive notes on it (link)
Integration of Google Forms, Google Sheets, App Scripts, and email automation.
Redux.
Using burp suite to monitor network calls in an Android emulator.
React Native localisation.
Background tasks in React Native.
Golang (basics)
Bash (basics)
Vim (basics)
Python (relearnt after losing touch).
Jupyter, Pandas and matplotlib.
Contract testing.

Reflections:

Your career (and your life) is the sum of luck, interacting with the right people, grabbing opportunities, and grinding out results. And the compounding of all of these over time. I realised it the most this year because of examples such as these:
- Working with a Tech Lead who not only is monumentally knowledgeable, but also gives many opportunities to his team members to shine.
- Working in a company that heavily emphasises on giving feedback, so that everyone is feeling good and self aware throughout the year.
- Working with a lot of smart people makes you look good too.
- Saying yes to one minor opportunity in the beginning of the year lead to my name being heard among some of the biggest folk in the company by the end of the year.
- My performance review was coincidentally postponed to after getting a word of praise from a client. So I knew in advance my review would go well.
Ironically, despite being a mere spectator to much of the success that you get, you still won’t get much success if you don’t work hard when it matters.
The power of saying yes to opportunities.
It lead to one benefit after another. Joining a volunteer group led me to learning about Google App Scripts and email automation. The work done with that team later led me to joining a global level team and meeting a few veteran TWers. It also got me working closely with an India-wide program manager, which gave me little bonuses like being featured in a newsletter and receiving great reviews.
Importance of appreciation.
TW brought the culture of giving appreciation in a big way this year, by building an internal application to express gratitude publicly to each other. For added measure, they gamified the experience to ensure maximum participation. Heartfelt messages that would’ve otherwise never been sent were pouring in from every direction. And the after-effect was a lot of people feeling good about themselves and their teammates, and happy to come to work the next day.
The power of being reliable.
Being the smartest person in the room can only take you so far (and my ego has been made painfully aware again this year that I’m not the one). It loses its value when you are not prepared to grind out results by any means imaginable.
In an industry where there is always more work than can ever be done, delivering on what you promised - being reliable - is gold.

My plan for 2022: DevOps, backend development and improving context switching as a team lead.

My ambition has evolved beyond being a technical 'expert'

Ray Mathew — Sat, 07 Aug 2021 16:47:15 +0000

Being in Thoughtworks, a rare company that treats its developers like human beings, has given me the luxury of stepping back from the fierce race of constantly learning the latest framework, reading its 100 page documentation, and discovering its intricacies in StackOverflow answers. Thoughtworks makes it very clear to developers that they are appreciated and encouraged to do a lot more than just write code to help fulfil the company's promises to clients. Developers are told to be leaders, manage stakeholder expectations, have expertise in security, conduct workshops for other developers, anchor communities, give talks in external platforms to promote the TW brand, work in social change projects that do not earn revenue, and so on.

This directive from TW has made me ask myself for the first time "Do I really want to be the guy who has in depth knowledge and years of experience in, say, frontend JavaScript? Do I really want to be known as the go-to guy for technical expertise in things like CSS, and hidden gems buried deep in a single line of OJET documentation?" My answer is No. And the clarity of my answer does not solely spawn from TW's support, but also things I learnt recently about product delivery, and the importance of doing things that matter in the larger scheme of life.

Product delivery

Product delivery has been broken down into a scientific and management endeavour over the last few decades. Research has been done, books have been written, conferences have been held, spin-off ideas have been spawned in multiple industries, and experts have arisen. And yet there still remains a creative aspect to it. You need to solve problems on the fly. You need to have people skills to notice which team members are producing poor results due to confidence issues rather than technical ineptitude. You need to focus on the most important problem when there are multiple fires you need to put out. For example, I've been having an itch for the last 3 months to write code to automate tracking of our Jira stories when a build reaches our QA environment, and send an alert to our Teams' group chat. Similar to Slack's Github bot. But I know that the more important problem is finding out why our automation environment sometimes has 17 random failures before succeeding, delaying our pipeline by 5+ hours. So I haven't scratched the itch yet.

Why we do what we do

Also, not enough people are able to connect the dots between what they're tasked and the larger goal of delivering a product that clients are happy with. If all developers only focused on technical tasks, while only the Technical Lead, Business Analyst, Manager and Quality Analyst focused on the business and functional logic, we're guaranteed to have a lot more bugs in our code. We're also guaranteed to miss our deadlines often - which affects our reputation, the client's bottom line, and our chances of getting more projects. Not enough people realise this. I do, and I care a lot about getting it right. It would be painful if a team of 100+ developers worked on a project for 1+ years, heroic moments were witnessed, all nighters were pulled, technical legends were praised, and at the 12th hour the client's end users saw a 502 error because there weren't enough integration tests.

As long as I'm able to contribute by helping more of my teammates realise this, I'm happy to step back from trying to be a technical wizard. I've anyway noticed that I'm not able to come up with clever bits of logic as fast as some teammates. I'll leave them to it instead of having a meaningless competition.

Disclaimer: I'm a complete proponent of doing things that have no value other than for the giggles. Just not when it comes to work.

What's the point anyway?

And finally, I've realised the importance of only doing things that will make a difference in people's lives. The world has so many problems, most of which cannot be solved in one lifetime. Add to it that there are millions of people creating more problems everyday, and it can feel like a suffocating, losing battle. In the midst of all this, what value are we adding for anyone, including our own, when we spend time on problems that don't make a difference? You may be busy, you may be seen as hardworking. But you are not making a difference in people's lives.

I'd rather be anyone but that person.

2020: A career review

Ray Mathew — Wed, 30 Dec 2020 09:24:30 +0000

Why am I bothering to write this?

I've been a software developer for close to six years now, but this is the first time I'm writing a review about the year gone by. Or writing anything at all related to my career. Many reasons prompted this change:

This has been a breakthrough year in multiple ways.
- My knowledge tree widened beyond what I thought was possible in such a short period, and yet the effort expended was not excruciating. In other words, the effects of compounding on previous knowledge have become as significant as acquiring more knowledge.
- I thought the learning curve I was on till the end of 2019 was roughly at a 60 degree angle - taking my work seriously, studying on weekends, solving others' problems on StackOverflow, and the occasional all-nighter. But when I started my new job at ThoughtWorks (TW) and saw how much people knew and learnt everyday, I realised my curve was closer to 20 degree. It really does make a massive difference when you surround yourself with the right people.
I realised this year the value of keeping records of every little detail - both professional and personal. The mind is truly a horrible hard drive; it retains only those memories which have strong emotions associated with them. This year's strongest emotional connection to work stuff are the two internal talks I gave. The stage fright was tantamount to mild panic attacks.
I've made a small adjustment in my career path due to the things I've learnt this year (explained below). I want to look back at this moment when I make my next adjustment.
I realised I would enjoy reading these kind of posts years later.
Unexpected benefits. If you expose your work to the outside world, sooner or later someone comes forward with a great opportunity, insight, or correction for you.

Milestones and unique moments:

Got a new job at the beginning of the year.
It's been surreal so far. I had no idea smaller companies of cultures similar to FAANG existed in India. I feel fortunate to have landed here, and bizarre that they reached out to me for an interview.
Went into a mental slump and loss of motivation due to my new salary and seeing younger people knowing much more than me.
I had been working for three years to make a big jump. At office I used to get my own work done and help my colleagues as well. Sometimes I would help up to four people at a time. On many days I would wake up at 5 A.M. and start answering Stackoverflow questions without freshening up. I worked on FreeCodeCamp projects and posted them on Github. I posted code snippets I was proud of on Codepen. I learnt design basics and CSS in depth, to became the 'go-to guy' for styling in my team. I wrote code to solve my friends' and my personal problems (Python scripts).

But, when I started my new job two things happened: 1. The three year goal I had been working on was finally complete. I felt empty and devoid of purpose. 2. I thought I was fairly knowledgeable after all the hard work I had done over the last 3 years - that thought shattered when I met my new younger colleagues.
Turbo-charged my note-taking and started tracking all aspects of work and personal life with Notion.
Gave 2 talks.
I gave 2 internal talks within the span of 4 months. My previous record had been to give 1 talk roughly every 3 years. The company's culture of sharing knowledge was definitely the biggest factor that pushed me to do this.
Stepped into security for the first time.
Last year, I had made a plan that I needed to have sufficient knowledge in all areas - backend, frontend, devops, security, delivery, finding clients, etc. before I could think about being my own boss years in the future. This year, I took the first, most difficult and important step by starting to learn about web application security. Why difficult? Simply because I had a mental block that it was difficult and a black box unlike any other.
First ever responsibility that spanned multiple teams - facilitating security assessments.
This milestone illustrates better than any other the theme of my transition from my old company to new, as well as my transition into a senior developer. In my old company my work was limited to creating UI pages, and I interacted only with the lead developer and manager. Facilitating security assessments, however, was my first ever task that involved multiple teams across the company. It was also my first foray into security. Interesting to note: these kind of roles are very common at TW - you can volunteer for them the moment you join as a college graduate. TW truly breeds leaders.
Read Extreme Programming Explained and The Phoenix Project, which massively affected my thoughts of what it means to be a developer, a team player, a consultant.
Extreme Programming taught me how to think like an agile team member, and contribute towards the common goal. It is a well-written book, and I would find myself wanting to improve things right away in my team every 10 pages. The Phoenix Project taught me to think even bigger - the goal of a company, the importance of a smoothly running pipeline, and the illusion of progress when you use bad benchmarks. I hadn't read many software-related books before these two. I'm excited and in awe of how much more knowledge I could gain in the future from similar books.
Switched to a MacBook after a lifetime of Windows.
What a change. I don't ever want to switch back. Except for maybe playing games.
Turned from a UI specialist to a generalist.
Partly my decision, and partly from the requirements of TW, I have decided to spend time learning backend languages as well.
Realized I wanted to be a big-picture person like Elon Musk, and not a specialist like Tesla's head of AI. Also, I value the importance of hiring the right people and having the right culture.
I'm not sure where exactly this change originated from. Maybe from reading the above two books, maybe from seeing how much my colleagues can go down the rabbit hole (spending 8 months trying to get one pull request accepted), maybe from seeing how my health starts to get affected when I delve deep into topics. Or maybe from realising more than ever that all your special skills are worthless if you're not providing value to the customer. I think the last one is the most important. Providing value can be done through simple things as well, and doesn't always have to involve in-depth knowledge. Sometimes it could be as simple as making old ideas better.

Also, from observing the culture of TW and the interview process (which includes questions about one's views on social issues), I realised how important it is to hire the right set of people in order to maintain high standards in a company.

Notable firsts:

Wrote my first co-authored post.
Wrote my first coding related blog post (this).
Used a web security testing tool (Burp Suite).
Started solving web security lab problems.
First ever responsibility that spanned multiple teams - facilitating security assessments.
Attended an online conference (NodeJS).
Gave 2 talks in a year.

New things learnt (unless stated otherwise, I learnt just enough to get work done. I can't be called the 'go-to guy' for any of these):

Docker
Kubernetes
ES6, ES2020
React
React Native
React performance tips from Kent C. Dodds
Chrome snippets
git rebase, filter-branch, stash, revert, gitignore_global
yarn
Jest, react-testing-library
Nginx
A ton of new terms, like canary testing, regression testing, smoke test, TDD, hallway usability testing, contract testing, broken window syndrome, desk check, Iteration Planning Meeting, timebox, spike.
Webpack
PWA
Testing for memory leaks in NodeJS.
How JS manages memory in the browser.
Log aggregation
CURL
Memoization
Tree shaking
How to make the browser and CDN cache data.
State machines
Web application security

It's never happened before that I do not have complaints for a full year. I truly couldn't have done anything better given the circumstances I was in, and not having the benefit of foresight.

My plan for 2021: Continue to learn about security and all other areas that would allow me to be called a generalist.

DEV Community: Ray Mathew

2024: Career Review

Milestones and moments:

Notable firsts:

New things learnt:

Reflections:

Building an API with Go, PostgreSQL, Google Cloud and CockroachDB

Links for easy reference

API Objective

The Domain Model

1. Setting Up The Local PostgreSQL DB

Installation

Create the user and permissions

Create the table

Add the data

2. Creating the Go Server

.env file

Middleware and Routes

Error handling

Request and Response structure files

Data Structure for the combination rules

Handler for GET \materia in api/handlers.go

In-server cache

Handler for POST \fusion in api/handlers.go

Swagger UI and OpenAPI definition doc

3. Setting up the remote PostgreSQL instance in CockroachDB

Data Migration

4. Deploy a Google Cloud Run instance

Using Google Secret Manager for the certificate

2023: A career review

Milestones and moments:

Notable firsts:

New things learnt:

Reflections:

2022: A career review

Milestones and moments:

Notable firsts:

New things learnt:

Reflections:

2021: A career review

Milestones and noteworthy moments:

Notable firsts:

New things learnt:

Reflections:

My ambition has evolved beyond being a technical 'expert'

Product delivery

Why we do what we do

What's the point anyway?

2020: A career review

Why am I bothering to write this?

Milestones and unique moments:

Notable firsts:

New things learnt (unless stated otherwise, I learnt just enough to get work done. I can't be called the 'go-to guy' for any of these):

Handler for GET `\materia` in `api/handlers.go`

Handler for POST `\fusion` in `api/handlers.go`