DEV Community: Sze Ying 🌻

My Git Workflow for Personal Projects

Sze Ying 🌻 — Sun, 27 Jun 2021 07:40:46 +0000

This post is targeted at all my friends out there who are looking to pick up Git. It should be a sufficient guide if you are looking to upload personal projects to code repositories like GitHub. If you are working on a project that requires collaboration with a team, do check out your team's docs on their workflow strategies instead.

What is a code repository?

An organised storage for your project's source code.

What is Git?

A version control system (source).

Why upload your project to a code repository like GitHub?

Version control - Code repositories like GitHub are built such that you can efficiently retrieve different versions of your source code.
Collaboration - Code repositories tend to have features like version control and code review which allow for ease of collaboration. If your code is open-source, it's also an invitation to people from around the world to comment and collaborate on your work.
Online portfolio - It's a place where you can showcase the work you have done.

My Git workflow

Here's how I most commonly use Git for my projects.

Creating a new project

1) Create a new project on the code repository website.

For GitHub, you can do so by clicking "New repository" on the right of the navbar, beside your profile picture

2) Upon creating your project, you should see instructions on how to link your local project up with the remote one.

The local project is the local version on your computer, while the remote project is what you uploaded to the code repository. It's similar to when you upload a file onto Google Drive. The changes you make in the local copy of your file will not be reflected in the Google Drive copy until you upload the file again. Similarly, if someone makes a change in the Google Drive copy, it will not be reflected on your local copy until you download the new file version.

Pushing updates to your project

1) Make changes on your local project.

2) To upload the new changes:
i. Add the changed files which you want to upload by doing one of the following:

// To add a single file /path/to/filename.js
git add /path/to/filename.js

// To add all files in the folder /path/to/
git add /path/to/*

// To add all files that have been changed
git add .

For MacOS users, you can download scmpuff, which will number off files that have been changed. You can then do something like:

git add 1-5

ii. Commit the added files

git commit -m "feat: added some new feature"

The last parameter in the above command is the commit message. Do write a meaningful commit message so you can easily tell what changes this commit consists of. The AngularJS Commit Message Conventions is a great place to start with how you can structure your commit messages.

iii. Push the new commits

git push

This step pushes your new commits to the remote project.

If you do try out this workflow, do let me know if you face any problems and I will update the content accordingly. If you are a comfortable Git user, I'd like to know how your workflow differs as well!

Set up State Management using React Hooks and Context API in 3 easy steps

Sze Ying 🌻 — Sun, 31 May 2020 08:31:20 +0000

I've been using React Hooks and Context API to do state management for all my React side projects. As shared in one of my previous posts, I first read about this approach in this blog post and found it very comprehensive and useful. With this approach, you can set up your state management in 3 easy steps:

Set up your Context
Provide your components access to your Context
Access your Context

The following code snippets assume we are writing a simple application that changes the colour of a circle according to user selection of the desired colour.

Choose thy color!

Step 1: Set up your Context

You can think of Context as a data store, while the Provider provides access to this data store to other components.

// src/context/ColorContextProvider.jsx

import React, { createContext, useReducer } from "react";
import { colorReducer } from "./color.reducer";

// Here we initialise our Context
const initialState = { color: "red" };
export const ColorContext = createContext(initialState);

// We use the useReducer hook to expose the state and a dispatch function
// These will provide access to the Context later on
export const ColorContextProvider = ({ children }) => {
  const [state, dispatch] = useReducer(colorReducer, initialState);
  return (
    <ColorContext.Provider value={{ state, dispatch }}>
      {children}
    </ColorContext.Provider>
  );
};

Personally I also choose to set up actions and the reducer separately to emulate the Redux lifecycle. Doing so makes it easier for my mind to understand how everything connects together.

// src/context/color.actions.js

export const SET_COLOR = "SET_COLOR";
export const setColor = (color) => ({ type: SET_COLOR, data: color });

// src/context/color.reducer.js

import { SET_COLOR } from "./color.actions";

export const colorReducer = (state, action) => {
  const { type, data } = action;
  switch (type) {
    case SET_COLOR:
      return { ...state, color: data };
    default:
      return state;
  }
};

Note about the reducer function: deep equality is not regarded in the detection of state change. There will only be detection if the state object has changed. Some examples:

export const reducer = (state, action) => {
  const { type, data } = action;
  switch (type) {
    case SET_PROP: 
      // State change will be detected
      return { ...state, prop: data };
    case ADD_PROP_TO_ARRAY:
      state.arr.push(data);
      // State change will not be detected
      // as the same state object is returned
      return state;
    case ADD_PROP_TO_ARRAY_SPREAD_STATE:
      state.arr.push(data);
      // State change will be detected
      // as a different state object is returned
      return { ...state };
    default:
      return state;
  }
};

Step 2: Provide your components access to your Context

To allow components to read or write from the Context, they must be wrapped with the Context provider.

// src/App.jsx

import React from "react";
import "./App.css";
import { ColorToggle } from "./components/ColorToggle";
import { Ball } from "./components/Ball";
import { Footer } from "./components/Footer";
import { ColorContextProvider } from "./context/ColorContextProvider";
import { Header } from "./components/Header";

function App() {
  return (
    <div className="App">
      <Header />
      <ColorContextProvider>
        <ColorToggle />
        <Ball />
      </ColorContextProvider>
      <Footer />
    </div>
  );
}

export default App;

Note that we don't wrap the Header and Footer components with the ColorContextProvider, so they would not be able to access the ColorContext. This differs from Redux's global store pattern where all components in the application can access any data in the state. By providing access to the state only to the components that require it, the modularity of state management is improved.

Step 3: Access your Context

There are two parts to accessing the context -- writing and reading. Both are done using the useContext hook.

Writing to the Context

For our simple application, we update the color value in our state every time the user clicks on any of the color toggle buttons.

// src/components/ColorToggle.jsx

import React, { useContext } from "react";
import { ColorContext } from "../context/ColorContextProvider";
import { setColor } from "../context/color.actions";

export const ColorToggle = () => {
  const { dispatch } = useContext(ColorContext);
  const dispatchSetColor = (label) => dispatch(setColor(label));
  return (
    <div className="toggle ma20">
      <ColorToggleButton
        label={"red"}
        onClickHandler={() => dispatchSetColor("red")}
      />
      <ColorToggleButton
        label={"blue"}
        onClickHandler={() => dispatchSetColor("blue")}
      />
      <ColorToggleButton
        label={"yellow"}
        onClickHandler={() => dispatchSetColor("yellow")}
      />
    </div>
  );
};

export const ColorToggleButton = ({ label, onClickHandler }) => (
  <button className="ma20" onClick={onClickHandler}>
    {label}
  </button>
);

Reading from the Context

We read from the state to decide what color to render the ball in.

// src/components/Ball.jsx

import React, { useContext } from "react";
import { ColorContext } from "../context/ColorContextProvider";

export const Ball = () => {
  // Again we use the useContext hook to get the state
  const { state } = useContext(ColorContext);
  return <div className={`ball--${state.color} ma20`} />;
};

And that's it! Just 3 simple steps and we have our state management set up. The full source code is here.

Do you use a different strategy for state management in your React apps? Please do share; I would love to try something different for my next side project 🍭

The "Chicken" Process: How we tackled a lack of Quality Ownership

Sze Ying 🌻 — Sat, 16 May 2020 09:38:46 +0000

A few months ago, we noticed that our sprint spillovers were constantly increasing in size. Upon taking a closer look, we realised that our existing workflow had an unhealthy emphasis on the number of merges into master. This emphasis was causing a false sense of development efficiency while introducing inefficiency in our delivery process.

The Problem: We left Quality Ownership to our Quality Engineer (QE)

A closer look at our agile workflow signalled that our Quality ownership was not evenly distributed.

	Delivery Stage	People involved
1	Story refinement and prioritisation	Product Owners, Devs, QE
2	Development of story on a new Merge Request (MR)	Devs
3	Code reviews	Devs, QE
4	Merge story into master branch	Devs
5	Ensure master pipeline runs to completion for story	QE

We use GitLab's CI service for our automated pipeline, which runs on every new commit to the master branch. It runs our unit and integration tests before deploying the new build to our QA and UAT environments.

In an ideal world, this would be the perfect setup — after all, everything was ~automated~. But here comes the problem: our integration tests were not stable. With this workflow, the burden fell on our QE to investigate and resolve each integration test failure. This task became exponentially harder with multiple MRs being merged at once by different devs, as well as the existence of undocumented flaky tests. It also didn't help that at that time, our devs to QE ratio was 5:1.

So many merges, yet not one green build...

This caused a bottleneck in our delivery process, which became especially visible at the start of each sprint, when we looked at the spillover from the previous sprint. Our spillover kept growing with majority of the stories stuck in the testing phase, but since the devs had "finished" all our work in the previous sprint, we continued to take up new stories.

We were now taking up new stories every sprint, but they were not being delivered to production at the same rate to create value for our users.

The "Chicken" process

Our old process went on for a while, frequently causing us to go weeks at once without a single green build. After many sprints of digging through numerous merges to debug pipeline failures, Jin Jie (the pained QE mentioned above) and Dickson (a developer who was helping our poor QE) from our team came up with the "Chicken" process. These were the rules for the new process:

One Merge at a time — Only one MR should be merged into master at any time, with the author of this MR being the "Chicken".
The "Chicken" owns the pipeline — The "Chicken" has to ensure that our CI pipeline runs to completion with a green build for their new merge. If there are any genuine integration test failures due to the new merge, open new MRs to fix them. The "Chicken" is free to open and merge as many MRs needed for a green build.
Ownership lies with the "Chicken", but Responsibility is shared by the team — The "Chicken" is not expected to fix all pipeline failures by themselves. If there are too many integration test failures, the "Chicken" is free to ask for help from the rest of the team.

Fun fact: The "Chicken" process was named after our residential chicken squeeze toy, which was used to get everyone's attention in the past. These days, it is a real life semaphore for merging into our master branch!

We have been trying out this new process for a few months now, have observed some significant benefits.

Quality is now a Shared Responsibility

Essentially, the goal of this new process is to eliminate the Somebody Else's Problem effect.

"The Somebody Else's Problem (SEP) field... relies on people's natural predisposition not to see anything they don't want to, weren't expecting, or can't explain"

— Douglas Adams in Life, the Universe and Everything

The "Chicken" process helped to mitigate the SEP field in our team with the following:

Greater visibility on our processes — Now that the definition of "Finish" includes ensuring that the corresponding merge passes the full pipeline, we are more inclined to learn what happens after merging our MRs into the master branch.
More context of the codebase — Investigating the codebase to fix tests gave us the opportunity to learn more about some of the long-lost contextual knowledge behind the implementation of older components.

Rather than having a single QE shoulder the quality burden, quality ownership was now a shared responsibility.

Greater Code Standards

We also saw an improvement in our code standards:

Greater focus on code quality — Having to experience the pain of investigating integration test failures, we have become more mindful of whether each MR we author will break existing integration tests. The old "merge and forget" mentality was now thrown out of the window; there is greater focus on writing reliable code and tests.
Discovering old bugs hidden in the codebase — With more effort in investigating test failures, we have also uncovered several bugs that were not caught during the development and review processes.
Fewer flaky tests — We spent a huge amount of time in investigating some test failures, only to realise that they were flaky. In other words, these tests would only sometimes fail, and the failures were largely due to the choice of testing strategy rather than incorrect implementation. This has motivated the team to work together to improve the overall reliability of our tests, rather than just clicking "retry" N times.

Improved Delivery Efficiency

Before the "chicken" process, we used to see 1-2 green builds per two-week sprint. Now, we are seeing 1-2 green builds every two days. On good days, we might even get up to 3 green builds in a day! 🚀 This was all possible due to

Resource reallocation to remove bottleneck — Rather than one QE investigating test failures for every single merge, the whole team was now invested to do so together. This allowed us to deliver stories at a rate closer to the rate at which we pick them up.
Debugging is easier when the diff is smaller — With only one merge going into the master branch at once, it was easier to inspect the diff due to the new merge. This made it a lot quicker to pinpoint where the failure point was when the pipeline failed on a new merge.

Our average number of green builds has increased!

Since adopting this new process, we have also made several modifications to it. One example is allowing multiple merges at once as long as there is still at least one person appointed as the "chicken". This modification helped us reap the benefits of the "chicken" process while also catering to our larger team size. We hope to continue improving on this process as our team's needs evolve.

Thank you Jin Jie and Dickson for your great efforts in improving our processes and also Joseph for providing feedback on this piece! Also thank you Jin Jie for providing the diagrams.

If you are now intrigued to work in a team like ours: we are hiring! Drop me an email @ ting_sze_ying@tech.gov.sg if you are interested to find out more 🌈

Programming your Telegram bot to fetch Document content with Telegraf API

Sze Ying 🌻 — Sun, 03 May 2020 15:29:39 +0000

While working on send4me, a message scheduler bot on Telegram, I ran into the question of how my bot can fetch the contents of a document uploaded by a user. I couldn't find a complete example for this use case anywhere, so I put together what worked for me.

A simple demo for fetching Document content

The following code snippet recreates the demo above. We will assume that you have already created a bot with BotFather and obtained your bot token.

const Telegraf = require('telegraf');
const axios = require('axios');

const BOT_TOKEN = ''; // TODO: get from BotFather

const bot = new Telegraf(BOT_TOKEN, {
  polling: true,
});

bot.start((ctx) => ctx.reply("Hello! Upload any document and I'll read it for you~"));

bot.on('document', async (ctx) => {
  const {file_id: fileId} = ctx.update.message.document;
  const fileUrl = await ctx.telegram.getFileLink(fileId);
  const response = await axios.get(fileUrl);
  ctx.reply('I read the file for you! The contents were:\n\n' + response.data);
});

bot.launch();

This should work for simple text files. For more complex files, response.data will need to be parsed before their content is usable.

I built a COVID-19 Visualisation with D3 and React Hooks

Sze Ying 🌻 — Sun, 26 Apr 2020 07:16:39 +0000

One Saturday morning after Singapore's Circuit Breaker began, I woke up thinking about this COVID19 visualisation. And I couldn't stop thinking about it. So I decided to build it.

I started with the crux of the project — the visualisation. The following is based on Singapore's COVID-19 cases dataset.

// App.jsx
import data from './data/graph.json';

const App = () => {
  const {nodes, links} = data;
  return (
    <div className="app">
      <Visualisation nodes={nodes} links={links} />
    </div>
  );
};

export default App;

// components/Visualisation.jsx
export const Visualisation = ({nodes, links}) => {
  const vizContainer = useRef(null);
  const width = 2048;
  const height = 1024;
  useEffect(() => {
    if (vizContainer.current) {
      const simulation = createSimulation({nodes, links});
      d3.select(vizContainer.current)
        .attr('viewBox', [-width / 2, -height / 2, width, height])
        .call((svg) => draw(svg, simulation, {nodes, links}));
    }
  }, [nodes, links]);
  return <svg ref={vizContainer} />;
};

To be honest, I am still struggling to understand how the following D3 code works. I didn't find many full JavaScript resources online (most were Observable notebooks) so I had to trial and error to piece everything together :")

// helpers/visualisation.js
export const createSimulation = ({nodes, links}) => {
  return d3
    .forceSimulation(nodes)
    .force(
      'link',
      d3.forceLink(links).id((d) => d.id),
    )
    .force('charge', d3.forceManyBody())
    .force('x', d3.forceX())
    .force('y', d3.forceY());
};

export const draw = (svg, simulation, {nodes, links}) => {
  // Defining links
  const link = svg
    .append('g')
    .selectAll('line')
    .data(links)
    .join('line')
    .attr('stroke', 'grey')
    .attr('stroke-opacity', 0.6)
    .attr('stroke-width', 3);

  // Defining nodes
  const node = svg
    .selectAll('.node')
    .data(nodes)
    .enter()
    .append('g')
    .attr('class', 'node')
    .call(drag(simulation)); // drag is some helper fn

  // Display nodes as images
  node
    .append('image')
    .attr('xlink:href', (d) => getImage(d)) // getImage is some helper fn
    .attr('x', -20)
    .attr('y', -20)
    .attr('width', (d) => (d.id.startsWith('Case') ? 50 : 100))
    .attr('height', (d) => (d.id.startsWith('Case') ? 50 : 100));

  // Add labels to the title attribute
  node.append('title').text((d) => d.label);
  link.append('title').text((d) => d.label);

  // This part updates the visualisation based on the current state
  // of where the nodes and links are. 
  simulation.on('tick', () => {
    link
      .attr('x1', (d) => d.source.x)
      .attr('y1', (d) => d.source.y)
      .attr('x2', (d) => d.target.x)
      .attr('y2', (d) => d.target.y);

    node.attr('transform', (d) => {
      return 'translate(' + d.x + ',' + d.y + ')';
    });
  });

  return svg.node();
};

After I was done with this part, I slacked off for two weeks. During these two weeks, Singapore's number of COVID19 cases shot up by thousands. With the large amount of data, my visualisation became incredibly slow. In fact, my laptop hanged 90% of the times I tried to load it.

For it to be usable again, I decided to add a filter component to filter the dataset by case number. For example, a filter value of 1000 would mean that only cases 1 to 1000 and their associated cluster nodes will be displayed.

I chose to use the react-rangeslider library. This might not be the best UX decision since the visualisation struggles with its load time, and a typical slider would mean multiple reloading while it is being dragged. Truth to be told, I thought a slider would be cool, and was too lazy to change it after realising it might be bad UX.

Anyway, to avoid the whole app freezing up due to the multiple reloading, I added logic to reload the visualisation only when the slider was no longer actively being dragged.

To pass the filter state around, I used React Hooks and Context API. Here's the comprehensive guide that I followed for this.

// components/CaseFilterSlider.jsx
export const SLIDER_MIN = 1;
export const SLIDER_MAX = 3000;

export const CaseFilterSlider = () => {
  const {state, dispatch} = useContext(store);
  const caseNum = state.caseNum;
  return (
    <div className="slider">
      <Slider
        min={SLIDER_MIN}
        max={SLIDER_MAX}
        value={caseNum}
        onChangeStart={() => {
          // store in our state that the slider is now active
          dispatch({type: SET_SLIDER_START});
        }}
        onChange={(value) => {
          // update the case number filter value in our store
          dispatch({type: SET_CASE_NUM, payload: {caseNum: value}});
        }}
        onChangeComplete={() => {
          // store in our state that the slider is now inactive
          dispatch({type: SET_SLIDER_COMPLETE});
        }}
      />
      Displaying {caseNum} cases
    </div>
  );
};

// updated components/App.jsx
import data from './data/graph.json';

const App = () => {
  const {caseNum, slider} = useContext(store).state;
  const [nodes, setNodes] = useState({});
  const [links, setLinks] = useState({});
  useEffect(() => {
    // slider is a boolean value to check if the slider was currently
    // active. This prevents a reloading of the viz before the slider
    // has reached its final value.
    if (!slider) {
      // filterData is a helper fn to filter out nodes and links 
      // relevant to the current filter value
      const filteredData = filterData(data, caseNum);
      setNodes(filteredData.nodes);
      setLinks(filteredData.links);
    }
  }, [caseNum, slider]);
  return (
    <div className="app">
      <h2>COVID19 - Singapore</h2>
      <Visualisation nodes={nodes} links={links} />
      <CaseFilterSlider />
    </div>
  );
};

export default App;

And that's all for the main logic of this mini project! It is still unusable for the full dataset — Singapore has 12,693 cases at the time of writing — so I defined SLIDER_MAX to only 3000. Perhaps an optimisation to cater for the large dataset could be to retain the old positions of nodes and links when reloading the visualisation. This could reduce computation time of the node and link positions when reloading the visualisation.

The full source code and dataset can be found here. Here's the live demo of this project.

Hello World

Sze Ying 🌻 — Sat, 25 Apr 2020 13:18:18 +0000

I've loved writing since I was a kid who was bullied in primary school. Every time I felt upset, I would write down everything I was feeling on a piece of paper, then tear it up and flush the pieces down a toilet bowl. It made me feel ever so slightly liberated from my then perceived problems.

The habit of jotting down my thoughts continued as I grew older. The medium with which I used to do so evolved from paper and pen to blogs, phone applications and 280-character tweets. Yet, I always stuck to writing about my personal life. The one thing I never did was to write about tech.

Why? I guess I don't think I have much to offer. There are so many people out there who are more expert at what I am doing, and I don't feel qualified to be writing about things that I don't 100% know about.

Today, I thought about the reason why I've kept up with writing about my personal life after all these years. I realised that I was never trying to educate anyone about anything. I never pretended to know more than anyone about anything. If I wrote something today, it would merely be to capture a snapshot of who I am today, so that I in the future can have something concrete to look back on, to reflect on my growth over time.

So I am letting that be my motivation to start writing about my work in the tech scene. I am not here to teach anyone anything, but I will share what I am learning today.