DEV Community: Samuel Ko

Idea Validation: ProofMate

Samuel Ko — Sun, 13 Apr 2025 07:58:51 +0000

Hi everyone!

I’ve been sitting on an idea and I’d love to gather early feedback from you.

The Problem

As a software engineer, the annual performance review is one of those tasks that always ends up draining more time and energy than it should.

Personal experience:

Year 1 - I dug through Teams messages, Pull Requests, Jira tickets, and more, trying to piece together what I’d done.

Year 2 - I started recording my "achievements" (with links) in Confluence throughout the year, but still had to look for extra details when writing my summary.

Year 3 - Same as Year 2, plus ChatGPT polishing before pasting it into Workday.

The Idea

I have this idea. Let's call it "ProofMate".

It will aggregate data from various platform for us, and generate a summary using AI.

PlantUML Text

@startuml
actor User as Engineer

package "ProofMate System" {
    rectangle "Data Collector" as Collector
    rectangle "AI Assistant" as AI
    rectangle "Self-Review Draft" as Draft
}

cloud "GitHub" as GitHub
cloud "Jira" as Jira
cloud "Confluence" as Confluence
cloud "Microsoft Teams" as Teams
cloud "Slack" as Slack

Engineer --> Collector : Authenticate & Connect Accounts
Collector --> GitHub : Fetch commits & PRs
Collector --> Jira : Fetch tickets & issues
Collector --> Confluence : Fetch documents
Collector --> Teams : Fetch chat summaries
Collector --> Slack : Fetch conversations

Collector --> AI : Send structured activity data
AI --> Draft : Generate draft review
Engineer --> Draft : Review & Edit

@enduml

Considerations

1. Privacy

Work data is often sensitive. Feeding raw material to AI models risks exposure. Should this be an offline-first app like jwt.io? More research is needed.

2. Data Retention

The app could be designed to store only the final summary (perhaps even save directly to your Google Drive). No long-term logging of your raw work artifacts.

3. Security vs. User Friendliness

Should API tokens be destroyed after each session?
How can re-authentication be made as smooth as possible?

Your thoughts

I’d love to know:

Does your company ask for self-reviews?
How do you usually prep for them?
Would you trust an AI to generate a draft based on your real work data?
What integrations would be a must-have for you?

I’d really appreciate your feedback! 🙏

(P. S. if you're also passionate about this idea, let's connect and build together!)

Secure Data Handling - Part 1 - Understanding Encoding

Samuel Ko — Wed, 09 Apr 2025 13:19:24 +0000

Have you ever entered a search term into a web browser, hit "Enter", and noticed that the URL changes to something like this?

https://example.com/search?query=hello%20world%21

The %20 and %21 represent a space () and an exclamation mark (!) respectively. They are part of URL encoding.

What is Encoding?

Encoding is the process of converting information into a different format, typically for storage or transmission.

A simple example is the "A1Z26" cipher, where each letter of the alphabet is replaced by its corresponding number (A=1, B=2, C=3, etc.). For instance, the word apple would be encoded as 1-16-16-12-5.

Practical Applications of Encoding

1. Base64 Encoding

Base64 encoding is one of the most common encoding schemes used to convert binary data into a text format. It’s often used to encode binary files (like images or attachments) so they can be transmitted over protocols that only support text (like HTTP, JSON, or email).

Example:
Plain Text: Hello, world!
Base64 Encoded: SGVsbG8sIHdvcmxkIQ==

Use cases:

Encoding files for email attachments
Embedding images directly into HTML or CSS

2. URL Encoding

URL encoding (also called "percent encoding") converts special characters into a safe format for URLs. For example, spaces and characters like &, =, and ? have special meanings in URLs, so they need to be encoded before transmission.

Example:
Plain Text: Hello world!
URL Encoded: Hello%20world%21

Use cases:

Encoding user input for GET requests
Encoding query strings or path parameters in URLs

3. HTML/XML Encoding

HTML or XML encoding ensures that special characters in data (like <, >, or &) are correctly interpreted by web browsers or other systems reading the data. For example, the character < in HTML needs to be encoded as < to avoid being mistaken for HTML tags.

Example:
Plain Text: x < 1
HTML Encoded: x < 1

This encoding is essential for displaying content within web pages without unexpected behavior.

Use cases:

Escaping special characters in HTML or XML

Encoding is Not Security

It's important to note that encoding is not a security measure.

Encoding is reversible. When you encode a piece of data, anyone can decode it. For example, with Base64 encoding, the padding (==) clearly signals the encoding method. Even if you invent a new encoding scheme, malicious actors can often identify patterns and reverse the process.

When working with sensitive information like passwords or personal details, you'll need encryption or hashing, which are designed for data protection.

Summary

Encoding is a simple yet essential concept in modern computing. It ensures that data can be safely and correctly transmitted across systems that might have different ways of processing it.

In the next part of this series, we’ll dive into encryption, which goes a step further to protect your data during transmission. Stay tuned!

Health Check for MySQL in a Docker Container

Samuel Ko — Thu, 03 Apr 2025 12:27:47 +0000

Glossary

Docker
MySQL

Background

When launching a full-stack web application, whether locally or in production, sequencing the dependencies is critical. For example, if we start serving web pages before the back-end services are ready, users may encounter broken links, 500 errors, or timeout issues that can be hard to debug.

Docker provides a built-in mechanism called HEALTHCHECK, which allows us to ensure that the database is ready to accept connections before exposing the REST API endpoints.

The Not-So-Perfect Way

One obvious way to health-check a MySQL database is to use mysqladmin ping.

services:
  mydb:
    image: mysql
    environment:
      MYSQL_ROOT_USER: root
      MYSQL_ROOT_PASSWORD: password
    healthcheck:
      test: mysqladmin ping
      interval: 1s

However, according to the official documentation, mysqladmin ping would:

Check whether the server is available. The return status from mysqladmin is 0 if the server is running, 1 if it is not. This is 0 even in case of an error such as Access denied, because this means that the server is running but refused the connection, which is different from the server not running.

It implies that mysqladmin ping is not a reliable way to check readiness. It could lead to intermittent REST API errors, which can be challenging to detect and diagnose.

We can validate such behavior by inspecting the Docker container:

The Better Way

Instead of simply pinging the server, a better health check is to send an actual SQL query to ensure the database is ready to handle requests.

test: mysql --user=root --password=password --execute="SHOW DATABASES;"

When inspecting the Docker container, we can observe the transition of the exit code from 1 (unavailable) to 0 (ready), indicating the database is now fully operational:

Conclusion

mysqladmin ping could return a false positive, indicating the server is running even when it's not ready to serve queries.
Use SHOW DATABASES; to verify the database is ready to accept connections and handle requests.

Have you encountered any issues with MySQL health checks in Docker? Or do you have any tips to share? Drop a comment below and let us know your experiences! We’d love to hear from you.

The Bug That Almost Made It to Production

Samuel Ko — Wed, 26 Mar 2025 13:27:10 +0000

In the fast-paced world of software engineering, there are two kinds of bugs:

Bugs that made it to production
Bugs that didn't
Bugs that were caused by "off-by-one" error

Today, I would like to share a story of #2 - one that was caught at the last possible moment.

The Setup: A Simple Clean-up Exercise

At a retail company with an e-commerce website, a junior developer was assigned a straightforward task: deprecate an unused field from an internal REST API.

How The Bug Was Born

As the junior dev navigated through 22 files to remove traces of the deprecated field, they spotted what looked like a "refactoring opportunity":

if (product.type === ProductType.BEST_SELLER) {
  doSomeThing();
} else {
  doSameThing();
}

Noticing that both branches seemed to do the same thing (or so they thought), they “cleaned it up” to:

  doSameThing();

But there was a tiny detail that went unnoticed: a subtle difference in function spelling.

And because this was legacy code, no unit tests covered this logic.

The Code Review That Didn't Catch It

By the time the pull request (PR) was ready, the developer’s IDE had also applied auto-linting across the touched files.

So, the PR ended up like this:

The result was, as expected, "LGTM".

The Disabled End-To-End Test

Meanwhile, another team maintained an end-to-end (E2E) test for the BEST_SELLER product. This test simulated a user placing an order on the website.

Unfortunately, this test failed intermittently due to data issues, such as the test product going out-of-stock.

So, when the test failed, the reflex action of the Quality Engineer (QE) was:

Disable the test first.
Investigate and rectify the data issue next.
Don’t be the person blocking a production release.

This time, however, something was different.

After disabling the test, the QE verified that the test product was actually in-stock, so they came to me (the software engineer) for help.

The Pressure to Release

It was a lucky day - both frontend and backend had production releases scheduled.

The QE team, having dealt with countless data issues in the past, was inclined to dismiss the issue and sign off on the releases.

But something felt off. The issue spanned across a range of products within the staging environment, not just the test product.

With pressure mounting, we escalated the issue to the Delivery Lead, who thankfully was understanding and gave us time to investigate.

This decision saved us.

By systematically comparing staging vs production behavior, we finally traced the issue back to the root cause — a tiny refactoring mistake hidden in a big PR.

Had we not caught it, customers trying to buy our best-selling products wouldn’t have been able to place orders.

Key Takeaways

1. One PR should do one thing only.

Mixing refactoring with unrelated changes makes it harder to review (and catch bugs).

2. When refactoring, ensure the changes are covered by unit tests.

If there aren’t any, write them first.

3. When enforcing a linting rule, reformat the whole codebase in the same PR as well.

The next person who raised a PR will thank you.

4. Never disable an E2E test without fully understanding the root cause.

A failing test might be the only warning sign before disaster strikes.

Table-Driven Testing in Typescript

Samuel Ko — Tue, 21 Jan 2025 10:22:15 +0000

Introduction

Table-Driven Testing, also known as Data-Driven Testing and Parameterized Test, is a popular methodology in Golang.

The idea is to run several sets of input data against the same logic, in order to increase test coverage. 💡

Let's look at an example in Typescript.

Example

The unit test:

describe('temperature converter', () => {
  it.each`
    celcius | fahrenheit
    ${-100} | ${-148}
    ${0}    | ${32}
    ${100}  | ${212}
  `(
    'converts $celcius degrees C to $fahrenheit degrees F',
    ({ celcius, fahrenheit }: { celcius: number; fahrenheit: number }) => {
      const result = convertCToF(celcius);
      expect(result).toBe(fahrenheit);
    }
  );
});

The test result:

Explanation

The test utilises Tagged Templates to run several sets of arguments against the same function.

If you're using VS Code, you can install Prettier to format the table for you. 🚀

Benefits of Table-Driven Testing

1. Reduce repetition

Table-Driven Testing not only reduces LOC in unit tests, but it also saves us from writing test descriptions for each of the test cases.

2. Enhance visibility

Table-Driven Testing groups the data at the top, enabling us to compare against the acceptance criteria. It also helps us to identify edge-case scenarios. 🔍

3. Enable integration with real-life data

It's possible to supply an array of data to it.each. It means we could potentially create an automated system that commit sample production data as a json file to the repository for unit testing purpose.

Below is an example of it.each using an array as input:

describe('temperature converter', () => {
  const testCases = [
    { celcius: -100, fahrenheit: -148 },
    { celcius: 0, fahrenheit: 32 },
    { celcius: 100, fahrenheit: 212 },
  ];

  it.each(testCases)(
    'converts $celcius degrees C to $fahrenheit degrees F',
    ({ celcius, fahrenheit }: { celcius: number; fahrenheit: number }) => {
      const result = convertCToF(celcius);
      expect(result).toBe(fahrenheit);
    }
  );
});

Limitations of Table-Driven Testing

1. Harder to debug

When one of the test cases failed in the table, we would need to comment out other test cases to debug.

2. Not as descriptive

A test case should be validating one particular scenario, e.g. positive value, negative value etc.

Without a meaningful description, it can be hard to understand the purpose of each test case.

To address that, we can add a text description column.

3. The table becomes too big

When a table has 4 columns of input and 1 column of output, it can become harder to understand the behaviour by purely reviewing the unit test. 😵

In that case, consider using smaller tables to describe individual behaviours, or refactor the function to receive less inputs.

A Bad Example

When we start using Table-Driven Testing, it's tempting to put every possible outcome into one big table.

And worse still, we may want to put conditionals into the test script. ❌

Consider the test below:

describe('temperature converter', () => {
  it.each`
    celcius | fahrenheit | error
    ${-300} | ${null}    | ${'celcius cannot be below 273.15'}
    ${-100} | ${-148}    | ${null}
    ${0}    | ${32}      | ${null}
    ${100}  | ${212}     | ${null}
  `(
    'converts $celcius degrees C to $fahrenheit degrees F or throws error $error',
    ({ celcius, fahrenheit, error }: { celcius: number; fahrenheit?: number; error?: string }) => {
      if (!!error) {
        expect(() => {
          convertCToF(celcius);
        }).toThrow(error);
      } else {
        const result = convertCToF(celcius);
        expect(result).toBe(fahrenheit);
      }
    }
  );
});

It has a few issues:

The table has holes. It means we have to consider null when writing the test.
if we need to debug a test case, we would need to determine which code path it takes within the unit test first.
The test script becomes much harder to read.

The better solution is to write a separate test for throws error if celcius is below 273.15.

Conclusion

Table-Driven Testing is a useful tool to increase test coverage. If used wisely, it can improve readability as well.

Thank you for reading.
Hope you find this post interesting!

Have you used table-driven testing before?
Share your experience in the comments!

A case study of creating configurable values

Samuel Ko — Fri, 17 Jan 2025 12:07:01 +0000

Glossary

DAO - Data access object

Case study

Once upon a time, there was a process within a monolithic enterprise system.

Its purpose was to receive product codes from external API, query the database, and send product data to another external API.

PlantUML Text

@startuml Example Process
skinparam componentStyle rectangle

database "Product Table"

rectangle "Process" {
  [Admin UI]
  [Service]
  [DAO]
}

[Admin UI] --> Service: clicks to run
[API 1] -> [Service] : product\ncodes
[Service] -> [API 2] : products
[Service] -> [DAO] : product\ncodes
[Service] <-- [DAO] : products

[DAO] --> [Product Table] : database query
@enduml

The new requirement

We want to divide the product codes into batches, so that the database query wouldn't be too big.

The proposed solutions

Obviously, there would be a value "number of products per batch".

1. The configurable solution

There is a school of thought - "Data should be separated from logic".

Following this thought, the value should be configurable from Admin UI, and subsequently passed down to DAO.

PlantUML Text

@startuml Example Process
skinparam componentStyle rectangle
skinparam defaultTextAlignment center

[Admin UI] as "Admin UI\n(with **value**)"
[Service]
[DAO]

[Admin UI] --> Service: value
[Service] --> [DAO] : value

@enduml

In this scenario, a few things could happen:

A system user accidentally updates the value to a non-sensible value, breaking the system.
Another requirement comes in to ensure that only developers can configure the value from the UI.
With hundreds of configurations, the system user would have a hard time finding the configuration that they can really update.

On the flip side, this approach does have an advantage:

We don't need to deploy code changes to adjust the value.

This approach is suitable for an environment where deployment is infrequent.

2. The environment variable solution

Given it's not an user-facing configuration, another approach would be to store the value in environment variable.

The DAO will directly obtain the value from environment variable and use it.

PlantUML Text

@startuml Example Process
skinparam componentStyle rectangle
skinparam defaultTextAlignment center

[Admin UI]
[Service]
[DAO]
[Environment Variable]

[Admin UI] -.> Service
[Service] -.> [DAO]
[Environment Variable] -> DAO : value

@enduml

It's a better approach because Service and Cron Job don't need to know anything about "Number of products per batch", which is purely a concept in application logic (as opposed to business logic).

In this approach, a few things could happen:

The code passes QA environment easily because the value is lower.
The code fails in Prod environment because the value is higher.
A lot of work is required to ensure environment consistency because some environment variables do vary between environments, such as API URLs.

3. The hard-coding solution

The value is only used by DAO.
The value should not change across environment.
Therefore, the best place for it is within the DAO itself.

It is version-controlled.
If someone changes it and breaks the system, we will know.

Conclusion

Configurable values mean maintenance and potential inconsistency.

Create configurable values only when there is a real need.

Thank you for reading.
Feel free to share your thoughts in the comments! 😉

Review your own pull request (PR)

Samuel Ko — Mon, 13 Jan 2025 10:36:41 +0000

Once upon a time, a developer created a PR and requested for review.

Little did he know:
❌ He misspelt a variable name.
❌ He forgot to remove a console.log("testing").
❌ He overlooked a // TODO: refactor this code.

As such, it took several rounds of review to get it across the line.

Had he reviewed his PR first, he could've avoided this disaster.

What is a PR?

A PR is basically a request of "May I put some code changes into your branch?".

In a collaborated environment, the main branch is usually protected. It means that you need to create a PR and obtain an approval to contribute your code changes.

Why should I review my own PR?

One may say, "I know my code and it works. Why should I review my own PR?"

To answer that, let me introduce a new made-up word - Approvability. 🚀

Achieving "It works." is the first step of creating a PR. Getting someone to approve it is the goal.

From a reviewer's point of view, it's much easier to approve an easy-to-follow PR than one that exhausts their brain power with ambiguous function names and obscure logic.

Apart from approvability, self-review can:

1. Eliminate stupid mistakes

Trivial mistakes like typos and code misalignment can make yourself look unprofessional.

2. Enhance readability

If you have a hard time reading your code, the reviewer will have an even harder time reviewing your code.

3. Improve code quality

When you review your own code changes from a bird's eye view, you may come up with refactoring ideas. 💡

How should I review my own PR?

Review it like a PR from someone else.

Ask yourself:

1. Does it satisfy the requirements?

When we're so focused to make the code work, we could miss some of the requirements.

We don't want the tester to come to us and say "It doesn't work!", right?

2. Does it exceed the requirements?

When we code, we may see things that we want to refactor (or fix). Very often, those refactoring are unrelated to the requirements.

It's called scope creep. The result is a PR with two sets of changes entangled together, making it harder to review.

When it happens, consider splitting the additional code changes to another PR. It will make your reviewer's life easier. ☺️

3. Do I need to think twice to understand the code?

If yes, it means that the reviewer will need to think 3+ times.

When it happens, we have 3 options:

Refactor the code to make it more readable
Use code comment to explain the "why" (but not the "what")
Use PR comment to invite discussions

4. Does the commit history make sense?

Reviewers may rely on commit history to follow your train of thoughts.

Before requesting for review, you have the opportunity to tidy up the commit history.

After requesting for review, do not force push.

Thank you for reading.
Hope you find this post interesting!

How do you review your own PR?
Share your experience in the comments!

An interesting observation on C# code coverage

Samuel Ko — Fri, 10 Jan 2025 11:58:49 +0000

Introduction

Code coverage is an indicator, not an metric.

In this post, I will share an example for your entertainment.

The code

public class Animal
{
    public bool CanTalk { get; init; }
    public bool CanWalk { get; init; }
}

public static class AnimalValidator
{
    public static bool IsHuman(Animal animal)
    {
        return animal.CanTalk && animal.CanWalk;
    }
}

The good test

Let's test all the input combinations.
It gives us 100% code coverage:

public class AnimalValidatorTest
{
    [Theory]
    [InlineData(true, true)]
    public void IsHuman_WhenAnimalCanTalkAndCanWalk_ReturnsTrue(bool canTalk, bool canWalk)
    {
        var animal = new Animal { CanTalk = canTalk, CanWalk = canWalk };
        var isHuman = AnimalValidator.IsHuman(animal);
        Assert.True(isHuman);
    }

    [Theory]
    [InlineData(false, false)]
    [InlineData(false, true)]
    [InlineData(true, false)]
    public void IsHuman_WhenAnimalCannotTalkOrCannotWalk_ReturnsFalse(bool canTalk, bool canWalk)
    {
        var animal = new Animal { CanTalk = canTalk, CanWalk = canWalk };
        var isHuman = AnimalValidator.IsHuman(animal);
        Assert.False(isHuman);
    }
}

The 100% coverage test

One day, someone removed two test cases ~~because they thought that the tests were running for too long~~ .

[Theory]
// [InlineData(false, false)]
[InlineData(false, true)]
// [InlineData(true, false)]

Interestingly, we still get 100% code coverage.

The explanation

Let's put the code into SharpLab.

This code:

return animal.CanTalk && animal.CanWalk;

will compile into:

if (animal.CanTalk)
{
    return animal.CanWalk;
}
return false;

It means:
CanTalk = true contributes 50%.
CanTalk = false contributes 50%.

The value of CanWalk doesn't matter.

The brittleness of 100% coverage test

Another day, someone changed the order of boolean ~~because they thought it would improve performance because majority of animals cannot walk~~.

public static bool IsHuman(Animal animal)
{
    // return animal.CanTalk && animal.CanWalk;
    return animal.CanWalk && animal.CanTalk;
}

Now, our test coverage drops from 100% to 50%, because both of our test cases are CanWalk = true.

Conclusion

C# code coverage is based on compiled code.

Writing good unit tests is more important than code coverage.

Hope you find it interesting.

An example of Consumer-Driven Development

Samuel Ko — Tue, 07 Jan 2025 07:55:25 +0000

It's my first time sharing my experience.
Let me know if there's anything I can improve.

Introduction

"Consumer-Driven Development" is a made-up terminology.

In this mindset, we define what the "consumer" needs first, then orchestrate the application to satisfy the need.

The "consumer" here can be perceived as the presentation layer, such as API controllers.

Example

The existing architecture

We have an application.
It calls an API endpoint periodically and logs the data:

The diagram is drawn using PlantUML.
PlantUML Text

@startuml Example App
skinparam componentStyle rectangle

cloud "API Endpoint"
cloud "Elastic"

rectangle "Application" {
  [Gateway]
  [API Client]
  [Repository]
  [Cron Job]
}

[API Endpoint] -> [Gateway]
[Gateway] --> [API Client] : HTTP Response
[API Client] --> [Repository] : Product DTO
[Repository] --> [Cron Job] : Product

[Cron Job] -> [Elastic] : Logs

@enduml

The new business requirement

The API endpoint provides a field called status.
It can be Approved, Disapproved, Pending.

We want to log the total number of Approved products.

Solution #1 - The data-flow approach

PlantUML Text

@startuml
skinparam componentStyle rectangle
skinparam defaultTextAlignment center

component [Repository] as "**Step 3**\nRepository"
component [CronJob] as "**Step 4**\nCronJob"

[API Client] -> [Repository] : **Step 1**\nProductDTO
[Repository] -> [CronJob] : **Step 2**\nProduct

@enduml

Step 1:
We want to obtain the data from the API endpoint.
Therefore, we create ProductDTO.Status as a text field.

Step 2:
We want meaningful data in our domain object.
Therefore, we create Product.Status as an enum.

enum Status
{
    Approved,
    Disapproved,
    Pending,
}

Step 3:
Let Repository do the translation.
(Note: Exception handling is omitted.)

var products = productDtos.Select(dto =>
{
    return new Product() { Status = Enum.Parse<Status>(dto.Status) };
});

Step 4:
Finally, we can log the number of approved products in CronJob.

logger.Information(
    "Number of approved products: {ApprovedProductCount}",
    products.Count(product => product.Status == Status.Approved));

Solution #2 - The consumer-driven approach

PlantUML Text

@startuml
skinparam componentStyle rectangle
skinparam defaultTextAlignment center

component [Repository] as "**Step 2**\nRepository"
component [CronJob] as "**Step 1**\nCronJob"

[API Client] -> [Repository] : **Step 2**\nProductDTO
[Repository] -> [CronJob] : **Step 1**\nProduct

@enduml

Step 1:
The consumer, CronJob, needs to know whether a product is approved.
Therefore, we create Product.IsApproved as a boolean field.

logger.Information(
    "Number of approved products: {ApprovedProductCount}",
    products.Count(product => product.IsApproved));

Step 2:
The provider, Repository, will figure out how to populate the new IsApproved field.
To do so, we create ProductDTO.Status as a text field.

var products = productDtos.Select(dto =>
{
    return new Product() { IsApproved = dto.Status == "Approved" };
});

Result

In solution #1, the enum value Status.Disapproved and Status.Pending are unused. We spent extra effort to handle exceptions when translating the text field into enum.

In solution #2, the use of boolean field makes the code much simpler. One could argue that it lacks data validation and scalability, but that is not within the requirement for today.

Conclusion

Using Consumer-Driven Development, we ensure that our effort is spent to meet the requirement, and nothing else.

Happy to hear your thoughts. 😉