DEV Community: Iman Tumorang

Hosting Internal Proxy and Internal pkg.go.dev for Internal Libraries Documentation

Iman Tumorang — Mon, 20 Sep 2021 05:12:21 +0000

Gopher reading docs and download library

Hey everyone, it’s been a while I haven’t published any blog posts. I think it’s been 6 months already haha. I was so busy for the past few months! There were just too many things happening and it kept me out busy from writing.

So here today, I’m sharing with you all about one of my experiments last year. Yes, it was last year. But hopefully, the content that I want to share here is still relevant and will help you with the pain I had previously in my job. The topic will be about hosting our own pkg.go.dev for our internal libraries at Xendit

Background

It’s not a secret, most of the companies have their own private libraries that will only be used for internal usage. It could be for internal logging, internal tracing, internal error libraries, or anything that could satisfy their systems needs.

It is the same for us at Xendit. My current team, DevPlatform, is responsible to help the engineer to be able to move faster. Thus, we want to help our product engineer work faster by providing and maintaining a lot of tools and libraries that are commonly used in every codebase. For instance, we built CLI Generator for some code templates, Logger libraries, and other common tools, both are written in NodeJS and Golang.

Specifically for Golang, we have various internal libraries for internal usages. In the beginning, we only had 2–3 Go libraries that were not hard to maintain. But as the team grows, the needs for Go libraries also grow bigger. We started to build more and more libraries until we realized, we already have a handful number of libraries written in Go.

Looking at the situation, we then observed a few problems that can cause the developer toil as the team growing. We want to solve these problems earlier so that our engineers will have a positive experience working with our Go internal libraries.

Problem 1: Looking for Documentation Website for Private Libraries

As a Gopher, I’ve been using pkg.go.dev since its first release. It’s a portal website for public Golang libraries. It really helps to see the detail of libraries without having to check the source code. It will generate the documentation site for any public libraries that are available in Github or any public Git provider (Gitlab, Bitbucket). So let’s say, I want to see what are the available functions in the Logrus library, I can easily see the complete functions in the pkg.go.dev like below.

Having realized that, I see that pkg.go.dev is only for public libraries. Yet for private/internal libraries, it is not available there. Because of that, I immediately try to figure out how we can host our internal documentation sites.

After looking for the solutions, I finally found pkg.go.dev website is already open-sourced here https://github.com/golang/pkgsite. That’s where we dig deeper in our exploration.

Problem 2: Easily Pull Private Libraries without Complex Configuration

Another problem we start to realize is, the engineering onboarding process to use Golang is quite longer compared to onboard them to use Node JS. For the Node JS project, we only need to create the .npmrc (a configuration for private NPM) then you will be able to pull all internal libraries.

But for Golang on the other hand, there isn’t anything like .npmrc for configuring the private libraries registry. Hence, it’s difficult, especially for new Gopher engineers. Sometimes, the error can be like this

$ go get [github.com/](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369)xendit[/internal-library-name](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369)

[https://sum.golang.org/lookup/github.com/](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369)xendit[/internal-library-name@v0.0.0-20190921175342-61a76c096369](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369): **410 Gone**

Or sometimes something like this

$ go get [github.com/](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369)xendit[/internal-library-name](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369)
# cd .; git ls-remote [https://github.com/xendit/i](https://bitbucket.org/bxcodec/math)nternal-library
**fatal: could not read Username for '** [**https://github.com'**](https://bitbucket.org%27/) **: terminal prompts disabled**

Even if you’re an experienced Golang engineer, solving this could be tricky. You either need to do:

Configure your local id_rsa to download from Github through ssh.
Setting your GOPRIVATE environment. So it will be pointing to the Github repo directly.
Or by using your personal Github Token to pull the private libraries.

And as a Devplatform engineer, we realized this is could be a problem. We then see that having a private proxy could help the engineers to download/pull their private libraries without the need to configure a lot of things in their local.

They will only need to:

Connect to our internal VPN
And set the GOPROXY environment

And they will be able to download all internal libraries.

And because of these 2 problems (easy to pull library and internal documentation site), we then decide to host our own internal proxy and documentation site in our own infrastructure.

pkg.go.dev Architecture

On the Github repo of the pkgsite, we can see how the pkgsite work. This is the complete architecture of how pkgsite works behind the scene.

pkgsite architecture from pkgsite repository

That’s quite complicated works. When I see this for the first time, I was overwhelmed, like, WTF! The time that I’ll spend on maintaining all of this is not worth the impact on the company.

Then I decided to spend a few weeks doing research on this in parallel. Whenever I was free, I tried to digest the code and the architecture, reading the docs, even asking many Go experts.

After having a lot of discussions with many people, I came to a basic conclusion. To host an internal pkgsite, we don’t have to deploy all those stuff on that diagram.

In fact, we only need the following two items

The frontend application > we can get it from the pkgsite repo
The Proxy server

The Frontend Application

The frontend app can be seen on the pkgsite repository. To deploy this is simple; since we’re using the Kubernetes, we only need to create the Dockerfile and the deployment and service config in the Kubernetes. Moreover, we need to register the domain to the internal domain.

This is how our Dockerfille looks like,

From the Dockerfile, as we can see, we pull the code of the pkgsite directly and compile it for docker distribution. Originally this Dockerfile is copied from Micah Parks’s Github repo (https://github.com/MicahParks/private-pkgsite). You might want to explore deeper, feel free to raise a PR or issue on that repository.

The Proxy Server

Since the pkg.go.dev is too huge to be hosted for internal needs, the only option we can think of is by using the proxy server.

We found a few candidates for the proxy server. But after trying it, we decided to use Athens as our proxy.

The reasons for this were:

It’s configurable, we can use custom DB, custom cache
It is used by a lot of companies. We believe it will be easier to debug and asking about issues in the communities.
The maintainers are active and a part of the Gopher Slack community. So if we found an issue, we can ask them directly.

To draw the architecture on how we use Athens from end-to-end, you can see the diagram below.

How we use internal proxy for Go private package

To deploy the proxy server turned out to be pretty fast, we only need to follow the documentation.

This is our Dockerfile example.

For the deployment, we deployed it on top of Kubernetes. Since our proxy is already dockerized, it helps us to deploy it to our infrastructure.

More documentation on how to deploy Athens proxy in the internal systems can be seen here. It may differ depending on the cloud/environment you use.

Security Concerns

Since the repo doesn’t have anything related to how to handle authentication. To avoid the internal libraries leaked to the public, the only solution we choose is by protecting the domain under an internal VPN.

Both documentations and Proxy servers are only accessible through VPN.

How to Use

For internal gopher engineers, if they want to use the Proxy, it will be really straightforward. Like I said previously on top of this article, they will only need to:

Connect to our internal VPN
And set the GOPROXY, and GONOSUMDB environment

$ go env -w GOPROXY="[https://proxy.golang.org,https://go-proxy.xendit.com,direct](https://proxy.golang.org,https://gomod.tidnex.com,direct)"

$ go env -w GONOSUMDB="github.com/xendit/*"

Some important notes:

The order for the proxy value in GOPROXY matters. From the example above, it means, the Go application will pull the library first from the public Golang proxy, then go to our internal proxy, and if the library still does not exist, it will go directly to Github.
When we useGOPROXY, we need to remove GOPRIVATE value. By default, the go application will look from the GOPRIVATE first. So the proxy won’t be used if the GOPRIVATE value is set.

After that, we will be able to download all internal libraries through the internal proxy.

Conclusions

After deploying both for Proxy and Documentation site(pkgsite) in our internal systems. We are now able to download internal libraries without complex setup in our local. Moreover, all of our private libraries are now available in our internal documentation site (pkgsite). The engineers are now able to see the documentation of all libraries with the same experience as any other public library.

The time that was needed to explore all these things took me around 1 month. Thanks to the infra engineer as well, he helped me to experiment on this. But I think if we do it intensively, it should take only a few days to work on this. I did not manage to finish it faster because I only worked on this during my spare time. So there’s no urgency and focus time ;). Although the priority was low, this documentation & library system is still helpful for future development and our internal Go ecosystem.

What’s next?

We’re planning to explore the same thing with TS/JS for our Node ecosystem at Xendit. The key is to reduce any developer toils as much as possible.

References and Thanks

Micah Parks (https://github.com/MicahParks/private-pkgsite)
Patrick Juen (our Infra engineer that helps me to explore this)
Stanley Nguyen for PRs review and design review

How I was performing Row Locking for Read-Write Transactions in Postgres

Iman Tumorang — Tue, 30 Mar 2021 03:43:20 +0000

It took me 2 days to attempt to solve this, but the solution is actually very simple and elegant.

Concurrent Acces by Xendit

Today's problem was complex with an obscure solution. I have been working on the refactoring of our core-payment system that handles the payment transaction to/from each bank that we support.

While developing this new core service, one of the services is related to generating a payment code, let’s call it the Payment Code Generator service. The payment_code needs to be random but also unique. In short, requires a counter to generate, which means we need to store the counter.

To simplify the scenario, let’s say I’m building a service that will generate a payment code.

In this service, I have 2 tables, master_counter and payment_code.

The master_counter table schema

BEGIN;
CREATE TABLE IF NOT EXISTS master_counter(
  id varchar(255) NOT NULL PRIMARY KEY, 
  user_id VARCHAR(255) NOT NULL, 
  counter bigint NOT NULL, 
  created_at timestamptz NOT NULL, 
  updated_at timestamptz NOT NULL, 
  deleted_at timestamptz, 
  CONSTRAINT master_counter_user_id_unique_idx UNIQUE (user_id)
);
COMMIT;

And the payment_code schema

BEGIN;
CREATE TABLE IF NOT EXISTS payment_code(
  id varchar(255) NOT NULL PRIMARY KEY, 
  payment_code varchar(255) NOT NULL, 
  user_id varchar(255) NOT NULL, 
  created_at timestamptz NOT NULL, 
  updated_at timestamptz NOT NULL, 
  CONSTRAINT payment_code_unique_idx UNIQUE(payment_code, user_id)
);
COMMIT;

In the master_counter, there is a counter column that will always increase based on received requests. If we draw into a diagram, the flow will look something like this:

Flow the Generating Payment Code

When the user requests a new payment code, we will get its current counter value.
The counter value is then increased by 1 and the algorithm is applied. (simply: hash the counter value to be a random character)
After the new counter value is hashed, the hashed value is stored to the payment_code table with the unique constraint.
The counter value is updated based on user id again to master_counter.
The hashed counter value is returned to the user. This is a payment code that will be used by the user.

While there are a lot of algorithms used to achieve this, the above flow sufficiently represents the simplified over all process.

Problems Started

Initially, when written in SQL, the query looks similar to this:

BEGIN;
SELECT counter FROM master_counter;

// Do Addition to counter in application
// Apply Generator Logic in Application

INSERT INTO payment_code(payment_code) VALUES(generated_value_with_counter);

UPDATE master_counter SET counter=<new_value_from_application>;

COMMIT;

However, I’m using GoLang, so there is native support for transactions. This can be seen in the example shown below:

tx,_:=db.Begin() // Begin the transaction

counter, err:= SelectCounter(tx) // Select the `counter`
if err != nil {
 tx.Rollback() // Rollback if any error occurred
}

counter++ // increase the counter
generatedCode:=ApplyAlgorithm(counter) // apply the algorithm

err:= SaveGeneratedCode(tx, generatedCode) // save the payment_code
if err != nil {
 tx.Rollback() // Rollback if any error occurred
}

err:=UpdateCounter(tx, counter) // update the counter to DB
if err != nil {
 tx.Rollback() // Rollback if any error occurred
}

tx.Commit() // commit the transaction

This function works perfectly when used as a single request. However, when load testing with only 2 concurrent users produced the following errors:

pq: duplicate key value violates unique constraint \"my_unique_payment_code_index\"

The “duplication error on payment_code” is happening because the generated payment_code must be unique.

While the application is behaving correctly by returning the error, from a business point of view, this is a problem. It should not be returning a duplication error, when the user is requesting a “ new ” payment_code that has never been used?

My first assumption is, if there’s a duplication, it means, that there’s a race condition here. The race condition occurred when I did the update and read on the counter value. The counter will be hashed for the payment code, and the payment code must be unique, but, because of the race condition, the application is reading the same value of the counter which is causing duplicate key violations.

In simple, the error occurs because we have met with a race condition where the counters are not updated when concurrent requests come in, causing duplication to occur when we attempt to create new payment codes in parallel requests.

First Attempt! — Using Isolation Level

Knowing this, I took to the internet to search and learn about the transaction, bring me to these articles:

I realized that there’s also an isolation level on the transaction. Reading all those articles made me remember my old days in college life. I remember I’ve learned this, but I had never found a case when I really needed to understand or apply this in practice.

From this research, I concluded that the isolation level on the transaction is not enough to lock the concurrent read-write-update processes in the database.

To address this, I added the isolation level to the application.

_, err := tx.ExecContext(ctx, "SET TRANSACTION ISOLATION LEVEL REPEATABLE READ")

if err != nil {
   return
}

After this fix, I re-ran the application, which returned a new error:

pq: could not serialize access due to concurrent update

This fix resulted in the avoiding the duplication error, but it also completely blocked concurrent requests. This is no different from having 1 concurrent request to a lot of concurrent requests.

Since from a business standpoint we need this system would be able to handle requests concurrently, I broadened my search to get advice from peers; colleagues, friends outside of work and even many Senior DBA from other companies.

Thanks to this crowdsourced effort of knowledge sharing, I realized, that the solution was row-level locking. Since I only need to read and update the counter value, we only need a row-level locking mechanism.

Second Attempt! — Using SELECT FOR UPDATE

Since I just want row-level locking, I didn’t need a strict isolation level because an isolation level is for table scope.

So my second try is:

Isolation level: READ COMMITTED (Postgres default isolation level)
Row-level locking: SELECT FOR UPDATE

The change is only adding the FOR UPDATE in my select query for my counter value. This new SQL query presents as follows:

BEGIN;
SELECT counter FROM master_counter **FOR UPDATE** ; // notice this

// Do Addition to counter in application
// Apply Generator Logic in Application

INSERT INTO payment_code(payment_code) VALUES(generated_value_with_counter);

UPDATE master_counter SET counter=<new_value_from_application>;

COMMIT;

This worked successfully, and in the end all it took was adding the FOR UPDATE to solve the problem. While this may not be the right end state solution, for now at least, it solves our current problem.

I discovered that there is a lot of things that you need to understand about SELECT ... FOR UPDATE — I found this article most helpful http://shiroyasha.io/selecting-for-share-and-update-in-postgresql.html

It’s amazing that this issue could be solved with only a simple SELECT ... FOR UDPATE. Maybe later there will be a new issue, but we leave that for our future selves 😎

Moving On: Building SEA Payment Infrastructures with Xendit

Iman Tumorang — Sat, 26 Sep 2020 03:03:02 +0000

Photo by Evie S. on Unsplash

It’s been a year after moving from Kurio. My past company. There are so many things I learned when worked in Kurio.

I learned about Products and Metrics pretty well. Learning many buzz-words in the product world like DAU, MAU, Retention Rate, Push Notification rates, etc. Data-Driven decision to create features in the product, and many things.

I also learned many Engineering practices. Like CI/CD, I learned how to make a CI/CD pipeline from scratch.

I learned about migrating from PHP into Golang. I learned about Microservice. Domain-Driven Design. API Driven Development, Software Testing. Semantic Versioning, Docker, Kubernetes, and many more buzz-word in Software Engineering. Creating Push Notification service, Logging service, profiling microservice, and many more.

I would say, the current me, I believe I’m pretty confident, I’m able to create a product from the backend side that will be needed. It’s thanks to Kurio.

I must say, it’s hard to leave Kurio. For the past year, I still meet my old’ friend there. Asking how Kurio’s doing. Is there any new issue, why is that? Is my legacy having bugs or issues? etc.

So, a few of my friends asked me, why do I leave Kurio? There’s a lot of reasons for that, but one thing for sure is, I really love scaling problem. I’ve talked to my manager about how I excited about handling big scale services. I’m hungry for more challenges.

Later I know, I’ve applied to many companies, I don’t remember which one. But mostly is on the scaling stage.

Moving to Xendit

Long story short, I finally choose Xendit. It’s not because I know someone here. To be honest, I don’t have any connection in Xendit back then. I don’t know what they’re doing. I even know what Xendit’s product when on the interview process. But I’ve known Xendit quite active in the community back then.

And, I like the vision of Xendit which is: Building SEA Payment Infrastructures. Infrastructures mean the foundation, the base. So, Xendit aiming to be the base of the Payment in the SEA. Not only Indonesia, but SEA, or event the world. And talking about payment, every business will have the payment stuff. So, we’re aiming to help every company from startup to enterprise to get paid and growing. We also want to born another unicorn born in SEA by helping them to handle their payment. So I’m not surprised, that many of our users are startups. And we want to grow together with them.

*above paragraph is a sponsored content 😂

And Xendit back then (till now as well) is still on the scaling stage. Like massively growing. I thought this is the right company for me to learn what’s the pain in helping the company to scale.

And this is also the reason (why) I’m not trying to apply to any unicorn company. The unicorns already too big! There’s no significant impact for me if I join them, except if I got a higher position tho, but I’m not confident yet for a higher position LOL. And looking for Xendit, I might still have a bigger impact, maybe bigger than the other’s company if I joined them.

Interview and Test to Xendit!

Photo by Van Tay Media on Unsplash

It takes 1 week for me to pass all of the tests. So in my period, we have 5 steps.

Screening with HR Manager
Pre-Assessment
Interview with Teams
Trials with the full team
Offered / Rejected

Screening is just a simple interview with the hiring manager. The questions are just about who am I, what’s is my previous job, and many more.

The next step is, obviously, the coding test. The HR give me the link for the test link. And it was a coding test. Luckily, the test for me is so easy 😂. I’m so lucky at the time.

After passing the pre-assessment, the next step is also a simple one. Interview with the teams. And no technical things here. Only about explaining things. And surprisingly, the interview was done in google hangout. So no coding on white-board or etc. So simple I think. I’m beginning to lose respect for this company. Why the test is just like that. 😂

But, the next step was the hardest part. They asked me to do a trial on their company for 2 days. So, then, I join the trials in their company. They give me a task to design some architecture of one of the payment channels in the company. I remember, they ask me to design a full architecture for BRI (one of Indonesian’s Bank) Virtual Account (VA).

In two days, I need to design a full architecture of the BRI Virtual Account, start from the Bank, and also how should Xendit handle it. With the limited time and resources, I tried my best, not to mention, I don’t have any experience in Payment or Fintech previously.

Not to mention, on the trial day, I also have an interview with the CTO. I’m starting to feel afraid, my respect point raised again to this company. Now I understand why the previous steps are too easy. I fell to the trap 😱

And later, on the final day of my trials. They asked me to present my design architecture to the team. Also, they asked me a few pseudo-code to do random VA generator. In a room, crowded by the teams from Software Engineers, QA, TechLead, and the Director Engineering also the HR. I’m starting to nervous and sweating 😂.

Photo by Headway on Unsplash

So much pressure in that room. Not to mention again, everyone is speaking in English. My English is not so good, so it’s really hard to understand what the hell they’re talking about there 😂 *no offense teams ✌️.

And then, I presented all of my findings of the BRI architecture. Also, so many questions come from the engineers. I don’t remember every question they asked. But I only remember the questions that come from the Director Engineering (to be my Engineering Manager later when I got accepted), that he asked me a few things about database and algorithm stuff.

Luckily, I passed that hell 😂. That’s was a 1-hour hell for me.

Challenges so far

So, now I’m working as Software Engineer in Xendit. Xendit is my third company after I graduated from 2016 ago. Starting for 6 months in Bornevia, and 2 years in Kurio, and now I’m here in my third year after I graduated from college.

First impression on Xendit? WOW! Just WOW!

So here are the challenges that I got for the first year.

Culture Shock!
Security and Policies!
Node JS world
Re-Architect and Re-Factor
Scaling Across Southeast Asia Region

Culture Shock!

It’s obvious, I face this one. I’m really shocked and surprised. From the culture, it’s really really different from my previous company. Here, we able to do remote works no limit (even without the covid)

Our teams located in 5 locations remotely, Jakarta, Singapore, Kuala Lumpur, Philippines, and also in the US, a few engineers work remotely from UK and Canada as well. And because of this, we need to utilize the Google Hangout pretty well.

So, sometimes, it’s hard for me to feel the flow. When I need to discuss a few things with someone, I must book their calendar. And if she/he in the office (Jakarta), we will be discussing it in the office. But if not, we have to do it through Google Hangout.

So, in the first week, I’m surely lost. I don’t know what to do. Because I still need to familiarize the culture.

Photo by Jonathan Rados on Unsplash

Not to mention, in my first week, everyone is really busy. I wonder why, but later I found that we have 2 issues and problems that need to solve faster and that issues affect all of the Xendit products.

So I’m totally lost. No one from the engineer can onboard me well. So, I must learn all of the systems by myself. When I’m stuck, I need to book one of the engineer’s time from the Google calendar. That’s really stressful for me in the first week.

Security and Policies

Another challenge that I got here compared to my previous company. Due to the needs for security/compliance, new technology should get buy-in from several stakeholders to make sure we can keep serving customers in an internationally secured way

Photo by Gayatri Malhotra on Unsplash

So it’s a bit hard to try new technology. Whatever the tools are, I can propose anything but it’s must follow the PCI (Payment Card Industry) Compliance that already standardized internationally. So it’s hard to find good tools and follow the PCI Compliance. There are so many things we need to check and follow.

Also, another challenge is, I need to do a PoC and create an RFC document which is pretty new for me. In my previous job, whenever I want to try new technology, I just implement it directly without do PoC and RFC things. Just do it! And deploy it!

Node Js World!

If the world created by a Programmer. Comic by Toggl.com

When I join Xendit, I come from Golang backgrounds. So I don’t have clear experience in JS. In my first company I use JS for 6 months, but I already forgot how to code in JS. The first challenge that I got here is only to re-learn everything. And it’s kind of really hard for me. Even till now, I’m still trying my best to learn Javascript and Typescript.

And after joining Xendit, it’s like a Deja Vu, and reminding me of my first company, everyone is freestyler and texas mode. Especially when building features and products.

But, even for that, I also trying to propose Golang here too. I even do a crash course about Golang here. And to be honest, it’s really really hard for me tho. Not only to evangelist the Golang, but I also need to help the team to make the flow more clear and standardized like to do refactor to the flows and the microservices.

Since right now everything is really freestyler, especially the engineering practice. So no wonder there are so many things that still need to improve here. And that’s what makes me more interested in joining Xendit. This is a scaling problem, and if we can solve this, then I believe Xendit will grow faster than the current conditions.

Re-Architect and Re-Factor

Another challenge is, in my second month, I’m proposing re-architecting of our current system for the payment services that handle the Virtual Account and Retail outlets.

Even I already have done so many re-factor, re-architecture, and re-writing services in my previous company. This is a different story. If in my previous company, if just only a small thing that makes the system complicated or not reaches our selfish standard, we just re-write it.

And later, when we have a new product, if the system is not good enough to support the new version, we then just re-write another service that has a similar function with it. At least this is what I fell when I work there. I’m not sure now.

But then, it’s possible because all of the services in my previous company are really well maintained and connected. The flow is clear from top to bottom, or vice versa. So we can easily do re-factor and re-writing.

It’s different from the Xendit has. Due to the large codebase and high interconnectivity between each service we need to think creatively when doing refactor to roll out a new system but making sure we are still able to serve the scale that we currently have to support all merchants.

Photo by Vikas Anand Dev on Unsplash

If only we do something changes to flow in service in a care-free, it can break everything. The services are so fragile.

So the solutions that we decide then is to build a new entirely flow of microservices, to avoid to break anything with the current productions. And the good thing is since I’m the one who proposes it, I make all the new services written in Golang, no particular reasons choosing Golang over Node, it’s just a matter of interest. Also, with some benefit, compiled to interpreted, it always compiled to be chosen in a matter of performances. Since the system that I proposed for re-architect is a core product, so I think, it’s good to say to choose the performance over the delivery time. We’re not rushing on development, like doing market fit, so it’s okay to choose Go over Node. Besides, the team approved it.

So, it really really hectic for me, at first, I need to deliver the new architecture, and also, I need to mentor the teams about Golang. This is totally a new challenge that I never faced in Kurio.

Scaling Across Southeast Asia Region

Another big challenge that I looking forward to in Xendit is about scaling to SEA Region, from Philippine, Singapore, Malaysia, and Indonesia, and other countries.

There are a few products that we want to replicate across SEA, like for example right now, Retail outlet payment that available in Indonesia and Philippine, Direct Debit payment, and many more.

And to support this movement, engineering problems will be very challenging in every aspect. From maintaining the legacy, then refactor to support multi-country. Security and compliance that can differ between country. It was totally a new level game for me.

So, sometimes I had a dream, which makes me like feels regretting, due to a bad decision to leave Kurio. But for now, I believe it’s not a bad decision, and I keep saying this to myself, I leave Kurio for the better of me, and to solve another level of challenges. And I’m looking forward to solving other challenges in the future wherever my journey ends :)

*PS, I’m hiring for my team, email me at iman[at]xendit[dot]co. In case you’re interested to join me to solve the scaling problem and many other challenges for building payment infrastructure in SEA

I drafted this after my probation ended in October 2019 last year, haven’t had time to check and publish until now. So I’ve updated a few words, like “been a year” or “past year” originally it was “been 3 months” or “past 3 months”. So when reading this, imagine it was still October 2019

Today I Learned: Golang Live Reload for Development using Docker Compose + Air

Iman Tumorang — Wed, 23 Sep 2020 11:20:40 +0000

Photo by Jeremy Perkins on Unsplash

Today, I’m trying to create a live-reload for my Golang application. It’s just a simple application of REST API.

For your information, Live-Reload is a mechanism that will reload our application on every file change. So it will keep up to date with your code. I’m not sure it’s the same as a hot reload. But people said live-reload is reloading the application when a file changes. And hot reload, will only refresh the changed file without losing the state of the application, so it’s not restarting the entire application.

Also based on this Stackoverflow answer,

Live reloading reloads or refreshes the entire app when a file changes. For example, if you were four links deep into your navigation and saved a change, live reloading would restart the app and load the app back to the initial route.

Hot reloading only refreshes the files that were changed without losing the state of the app. For example, if you were four links deep into your navigation and saved a change to some styling, the state would not change, but the new styles would appear on the page without having to navigate back to the page you are on because you would still be on the same page.

I guess, for this article, I can say this a live-reload because I’m not sure we’re able to use hot reload for Golang. Because for Golang, for every change, we need to restart the application from re-compiling it and re-run it. So the most possible way to do it is to use live reloading instead of hot reloading.

Background

But before telling the details, I want to talk about the background, why I just learned this live-reload after so many years?

I’ve been working with Golang for 3 years now, but I never use live-reload when working on my projects. It’s not because I’m an ignorant person, I do want to use live-reload for development.

But for the Golang case, I have a few reasons back then, why I haven’t used it.

Golang is a compiled programming language. My mindset to think like it’s impossible to have live-reload for compiled programming language. Because we need to compile the application and run it. Well, technically I wasn’t wrong, even we can like to create a live-reload tool, to watch the changes and re-compile the application (in away, it’s live-compile-run)
Golang run/build is quite fast compared to any compiled programming language in the market, also one of my reason not to consider live-reload yet, at that time.
Making my own live-reload tools. Well, it’s just reinventing the wheel, not to mention the time that needed to build it is not worth it, back then. Even sometimes when I’m working a larger project, I’ll feel regretted not to started making my own live-reload tools.
I’ve tried a few tools but disappointed by the performance. Actually, I’ve tried once back then, I forgot what’s the tool, but back then, it consumes a lot of my laptop resources. It making my laptop lagging. Been trying to some frameworks as well that have a live-reload feature, but again it consuming a lot of resources every time I save my file. Instead of increasing my productivity, it’s making me more stressed because of the lagged laptop.
My projects relatively depend on small dependencies (libraries, framework, database, any extra layer), so compiling and running the application is not an issue yet at that time.

So then, now I’m looking for a live-reload feature because my projects getting bigger and also the dependencies as well. Like it depends on Google Pubsub, Firebase, Mongo, Postgres, Redis, all other libraries. Just to start the app, it will take time. So when developing new features, it will really take time only to run it locally. That’s the time when I try to consider to use live-reload for my productivity. Well at least, reducing the time needed for manual intervention like compile and building the application. Even milliseconds matter when you are in zone LOL.

So then, I have a few criteria for the live-reload,

The most important is, it’s not making my laptop lagging. Performance is the most important, I don’t want to get stressed by a lagged laptop. I got a bad experience when using one of the Go frameworks that have a live-reload feature, it frustrates me. In the beginning, it just works fine, but later when I make frequent changes, it made my laptop lagging like a hell. Even when typing in the editor, it’s got delayed.
Configurable! I want those tools can be configurable with my own settings for the live-reload.
Easy to use and portable. So everyone can use it no matter what’s the environment. Generally speaking, I would say, it’s dockerized at least. Since I know docker has become a standard tool for engineers now.

And based on that criteria, I tried to find many tools on the internet. I found a few good tools, but then, I remember, my colleague use a live reload for our internal project in the company.

Out of curiosity, I then try to explore that tool. It’s named, “Air”. The repository can be found here, Air.

Live Reload With Air

The first impression, I feel skeptical, because it’s just like yet another live-reload tool. I have to install it, then run it in my project. But, when I check the Github repo, the winning solutions that they provide is,

Configurable. I can configure my settings based on my need.
Portable. Since it just a binary, I can use it to docker, and make it portable.

Setting Up My Live Reload Environment

To use the Air live-reload tools, you see in the Github repository. But for me, I use Docker compose to manage my live-reload development environment.

So what I will need to have,

Application the I will develop.
Installed docker on my laptop.
Dockerfile for development.
Docker-compose file for development.
Customize the configurations

So the idea is, I will use docker-compose to manage the live-reload with using Air.

Live reload idea with Docker compose + Air

1. Making the Dockerfile for Development

The first step is to make the docker file for development.

The docker file is basically only to download and install the Air binary and make it as the entry point for the docker.

If you see from the above docker file, you will see this line

RUN curl -fLo install.sh https://raw.githubusercontent.com/cosmtrek/air/master/install.sh \ && chmod +x install.sh && sh install.sh && cp ./bin/air /bin/air

This is only to get the installation script, then run it, and making it /bin folder.

After that, then we will make the docker-compose file for the development.

2. Making the Docker-Compose file for Development

The next step is to create a docker-compose based on the previous dockerfile. As you can see, in that docker-compose file, I set the dockerfile to dev.Dockerfile

web:
    build:
      context: .
      dockerfile: dev.Dockerfile

Also, the other important thing is, I set the volume link to my current directory,

volumes:
      - ./:/app

It means the container will use my current directory and attached it to /app in the container. So If there are any changes in my directory, it will also change the file in the container.

3. Set your application config

Since we will do a live-reload, so, we also need to provide a config file for our application to be able to run. The config file can be different, usually, people use ENV variable or .env file. Or just a config file, like config.jso n or config.toml etc.

You need to define it if you want to make a live-reload the application. Just assume you will need to run the application, what config you need.

In my case, on my local, I will create a file named config.toml

title="Configuration File for Menekel"
debug=true
contextTimeout="2"
[server]
  address= ":9090"
[database]
  host="mysql"
  port="3306"
  user="root"
  pass="root"
  name="article"

And this config will be copied to the container as well, so we will be able to run it in the container.

4. Set the Air configurations

The next step is, create the Air config. Create a new file with named .air.toml and then configure it based on your needs. I’ll try to break down any important configuration terms from the config here.

a. CMD syntax

[build]
//...
cmd = "go build -o ./tmp/app/engine app/main.go"
//....

The bold one from above is for the command that you need to compile your application. In my case, I just use go build -o ./tmp/app/engine app/main.go . So if you guys want to copy this to your application, make sure the build command is correct.

b. BIN syntax

[build]
//....
bin = "tmp/app" 
//....

The bold text above is only the directory of the compiled binary exist.

c. FULL_BIN syntax

[build]
//...
full_bin = "./tmp/app/engine http"  
//...

The next one from above is how your application will be started after compiled. In my case, to run the compiled application I will need to pass the argument http to determine I want to run the HTTP server.

So if your application doesn’t need any argument, you don’t have to add any additional arguments.

If your application using the ENV variable, you will also need to pass it like,

full_bin = "ENV1=mysql ENV2=localhost . /tmp/app/engine http"

Tricks:

Make sure your application to be able to read .env file, so you don’t have to list all the ENV in the Air config.

Use a config file, but bare the consequences.

d. INCLUDE_EXT and EXCLUDE_DIR syntax

[build]
//...
include_ext = ["go", "yaml"]
exclude_dir = ["tmp"]
//...

The include_ext will watch for every file that has an extension as listed. If there are any changes in any file in the project, it will trigger Air to re-compile the application.

The exclude_dir is to tell, any changes on that directory won’t trigger the Air to recompile the application.

e. DELAY syntax

[build]
//...
delay = 1000 # ms
//...

Sometimes, we may make too frequent changes, so to reduce the instant build every saved file, we can add a delay. So it won’t directly compile the application for frequent changes. This one is pretty helpful since compiling and re-running the application will consume the CPU, if we compile to frequent it will make our laptop lagged. And with this, we can configure the delay to reduce the pace for compiling the app.

5. Final Steps: Making It All Running.

The final step is, you can run the application by using docker-compose,

$ docker-compose up -d

And check for the log in your terminal,

$ docker-compose logs -f

See the demo here,

Conclusions

After trying this, at least I can get work faster. If the changes are too frequent, I just need to add the delay in the config, so it won’t affect my laptop resources.

The example repository can be seen here, Menekel, or you check this PR if you want to see what changes I made for adding the live-reload.

Also, if you guys have any better ideas, tools, let me know, put in the comment below so the other people also can try it.

If you guys found this useful, like, and share so other people can reach this, spread the love, and knowledge.

How to Add Your Recently Published Medium Articles to Your GitHub Readme

Iman Tumorang — Sun, 16 Aug 2020 16:32:37 +0000

Show off your latest Medium work on GitHub

Photo by Christin Hume on Unsplash.

GitHub recently released a new feature that allows you to create a Readme profile, so you can now customize your GitHub profile page.

You can see an example on my GitHub profile:

My GitHub profile.

This feature is really nice. It makes your GitHub profile look more professional and content-rich. In the future, I expect GitHub to look like LinkedIn for developers.

Introducing GitHub Readme — Recent Medium Articles

I’ve seen a lot of plug-ins that people have made, like the GitHub stats card, programming language stats, and even games (e.g. this online chess game or even tic-tac-toe).

In this readme, the user has included a list of their recently published Medium articles. But it’s only available on their profile. To achieve the same results, I need to copy their code, which takes time.

That’s the original idea behind this plug-in. I created a separate repository with a customized function. I then made the function more generic so everyone can add their recently published Medium articles to their GitHub readme.

Steps

To use this plug-in, you only need to add this script to your GitHub readme:

https://medium.com/media/60fa24f400869923cd2f970850db9bf1/href

So the format is:

https://github-readme-medium-recent-article.vercel.app/medium/<medium-username>/<article-index>

medium-username: Your medium username/profile
article-index : Your recent article index (e.g. 0 means your latest article)

The full steps can be seen in my repository. Also if you’ve found any issues, just open an issue or create a PR directly on that repository.

More About This Plug-In

I’m using Vercel for the static hosting and the serverless function to retrieve the recent articles. I might be able to add a custom domain, but that’s for later.
I’m using an RSS feed from Medium. You can get your RSS feed from Medium by entering this URL: https://medium.com/feed/@yourMediumUsername.
Then convert it to JSON using API RSS to JSON:

[https://api.rss2json.com/v1/api.json?rss\_url=https://medium.com/feed/@imantumorang](https://api.rss2json.com/v1/api.json?rss_url=https://medium.com/feed/@imantumorang)

Make it generic so everyone is able to use it.

To make it generic so people can directly pass their Medium username to the serverless function:

https://github-readme-medium-recent-article.vercel.app/medium/@imantumorang

I need to make the folder of my serverless function as follows:

└── medium
    └── [user]
        └── [index].ts

The [user] directory is required so I can make the username dynamic. I was stuck on this problem when I was making the plug-in. Actually, I can create it using query-param like ?username=@imantumorang, but from my experience and keeping REST in mind, making it path-param is the proper way to tell that the param is required. Also, I want to keep the experience the same as when you’re visiting your Medium profile (e.g. medium.com/@imantumorang).

I knew that comments in the Medium articles would be added automatically to the RSS. To only display the article, I added a filter function on the function:

if (thumbnail.includes("cdn")) {
        fixItem.push(element)    
}

So I only enabled articles that have a thumbnail. I’m still looking for a workaround for this because if the article doesn’t have any thumbnail, it will be skipped. At least for now, if your article has a thumbnail, it will be displayed when we import it to your GitHub readme.

It may take time to be able to get your recent article to be listed (for a new article) since the API RSS to JSON is cached. Please wait around 1-3 hours after the article has been published on Medium.

Conclusion

Well, I think that’s all for now. If you find any issues, you can directly open an issue on GitHub. I’ll try to help as much as I can.

How To Do Pagination in Postgres with Golang in 4 Common Ways

Iman Tumorang — Tue, 26 May 2020 05:06:05 +0000

A few examples of pagination on Postgres, with Benchmark, written on Golang

Photo by Ergita Sela on Unsplash

Hi again everyone, it’s been a long time I haven’t published any article. There’s a lot of things happen, like from the pandemic and much more stuff. This pandemic affecting me mentally personally, like this self-quarantine is really exhausting and stressing me enough. I wish this Covid-19 pandemic will be ended before Christmas this year. 😭

On this rare occasion, after fighting with the boredom and laziness, I found a spirit to finish this article. Start from me when building our new application in my current job, I’m curious about a few things, and in this part, it’s about pagination. Like how to handle pagination in a better way based on my understanding LOL. *My proposed idea might not be the best, so if you guys have a better way, a vast experience than me, put your comments below yaa!!

TBH, I never care more details about this in my previous job because we all have the same perspective, and we only like to have 10 engineers in my previous company, so we can have the same perspective. But now I care about this, since we have a lot of engineers in my current job, and everyone has a different perspective.

So, I’m just curious, what’s the better way in building pagination on Postgres on top of the application, with my case I’m using Golang for the application.

Actually, there are 2 famous styles of pagination:

Cursor based pagination
Offset based pagination

In this article I’ll only cover those 2 style paginations in 4 different common ways that Backend engineer usually does, or at least, what I know so far since I know how to code.

Do pagination with page number, pretty common, the user only sends the page number, and we handle it internally, I use offset in the database level.
Do pagination with offset and limit, pretty common since the RDBMS features. The user will directly send the offset number from query param.
Do pagination with a simple query with an auto incremental ID as the PK, quite common for auto incremental ID in the database. Which is the ID is treated as the cursor.
Do pagination with UUID as the PK combined with the created timestamp, also known as the seek-pagination method, or keyset pagination method. And the combined key will be hashed into a cursor string.

So what I’m gonna do here are, I’ll create those 4 pagination implementations, and do a small benchmark from code, I’ll using Golang Benchmark. The goal of this article is just to satisfy my curiosity LOL. I know I can read people’s articles, but I want to do it with my own version.

TL;DR

All the code used here already pushed to my Github repository, github.com/bxcodec/go-postgres-pagination-example
Conclusions can be seen at the bottom of this article

Pagination On REST API

To give you some context, *in case you don’t know what is pagination used for. Pagination is used to paginate your response, LOL. Well, I don’t know how to rephrase it better.

I’ll create an example, let’s say I have this endpoint, in REST API.

GET /payments

And this endpoint will fetch all payments from the API. As we know, in bigger scale application that has tons of data set, these payments may have thousands or millions of data rows. And as a user, I want to fetch my payments list.

From a database perspective, querying all the records will takes time a lot. I can imagine how long it will be if we have a million records and fetch all the data. So, in that case, people introduce what they called pagination. It works like pages on the books, that each page contains a bunch of words.

But for this endpoint, each page will contain a list of payment details, so we can still fetch the payment faster but maybe it will truncated into multiple pages until we can fetch all the payment records.

GET /payments?page=1 // to fetch payments in page 1
GET /payments?page=2 // to fetch payments in page 2
GET /payments?page=3 // to fetch payments in page 3
... etc

You may have seen this style in any endpoint, or maybe something like this as well.

GET /payments?limit=10 // initial request for fetch payment
GET /payments?limit=10&cursor=randomCursorString // with cursor
GET /payments?limit=10&cursor=newrandomCursorString // for next page
GET /payments?limit=10&cursor=anotherNewrandomCursorString
... etc

And many more, this is what we called pagination. We truncate our list of data into a few segments and send it to the client, so we still maintain the performance of the application and the client won’t lose track when fetching our data.

1. Pagination with Page Number

GET /payments?page=1 // to fetch payments in page 1
GET /payments?page=2 // to fetch payments in page 2
GET /payments?page=3 // to fetch payments in page 3
... etc

Have you seen pagination like those above? TBH, I never have seen any pagination like those, not in public API if I remember correctly. But, I’ve ever created pagination with that’s style, around 4 years ago, on my first job-test after graduated.

So the logic quite complicated in the backend, but it will simplify from the user experience,

First I’ll set the default limit, let’s say 10. _Per page is 10 item_s.
And each page number will be multiplied to the default limit
Then I’ll use it as the offset to the database.
And, the user can fetch the items based on the requested page number.

So then, I try to build again a simple application for this kind of method. With 100K rows of data , I try to benchmark it.

Benchmark Result

The drawback of this pagination method is

Performance-wise, it’s not recommended. The bigger the data set, the bigger the resource consumption.

But the benefit of using this method, the user feels like opening a book, they will just need to pass the page number.

2. Pagination with Offset and Limit

Pagination with offset and limit is quite common to engineers. This comes because of the feature of RDBMS that supports offset and limit for querying.

From the application level, there’s no extra logic, just passing the offset and limit to the database, and let the database do the pagination.

How is usually looks like,

GET /payments?limit=10 // initial 
GET /payments?limit=10&offset=10 //fetch the next 10 items
GET /payments?limit=10&offset=20 //fetch the next 10 items again
... etc

From the client-side, they only need to add the offset params, and the API will return the items based on the given offset.

And from database level, which is RDBMS, it will look like this below,

SELECT
    *
FROM
  payments
ORDER  BY created_time
LIMIT 10
OFFSET 20;

Benchmark Result

The drawback of this pagination method

Performance-wise, it’s not recommended. The bigger the data set, the bigger the resource consumption.

The benefits of this pagination method

Very easy to implement, no need to do complex logic things in the server

3. Pagination with Auto Incremental PK of the ID

This pagination method was also pretty common. We set our table to be auto increment, and use that as the page identifier/cursor.

How it’s used in REST

GET /payments?limit=10
GET /payments?limit=10&cursor=last_id_from_previous_fetch**
GET /payments? limit=10&cursor=last_id_from_previous_fetch
... etc

How it looks like in database query level

SELECT
   *
FROM
  payments
WHERE
  Id > 10
LIMIT 20

Or for descending

SELECT
   *
FROM
  payments
WHERE
  Id < 100
ORDER BY Id DESC
LIMIT 20

Benchmark Result

The drawback of this pagination method

The only drawback of this pagination method is, when using the auto-increment id, it will be problematic in the world of microservice and distributed system. Like id with 20 can exist in Service Payment and Service User. It’s unique in the same application context. It will different if each ID using UUID, it’s “practically unique” (means, there’s a very small possibility of duplicate generated UUID). So some people trying to use UUID instead as the PK. Read more details about UUID and auto-increment keys here

The benefits of this pagination method

Easy to implement, no need to do complex logic things in the server.
The best way to do pagination that I know so far from performance-wise, since it’s using autoincrement ID.

4. Pagination with UUID Combined with Created Timestamp

I’m not sure this is pretty common, but I see that a few articles do this kind of pagination. The context is, the table not using auto incremental id, but it’s using the UUID instead. But then people wondering how to do pagination, adding a new column with auto incremental number is a wasting resource. So for myself, what I do is, using the created timestamp of my rows, and combine it with the PK which is the UUID.

This is the database schema

And for the faster queries, I make an index with multiple tables which is the PK and the created timestamp, as you can see from the above schema, I made an index named idx_payment_pagination.

So the logic is,

I’ll use the UUID which is my primary key, and combine it with create timestamp
Combine those two into a string, then I encode it to base64 string
And return that encoded string as a cursor for the next page, so the user can use it to fetch the next page of their request.

Example of how I made the cursor on application level

And this is how it looks like in the REST endpoint.

GET /payments?limit=10
GET /payments?limit=10&cursor=base64_string_from_previous_result
GET /payments?limit=10&cursor=base64_string_from_previous_result
... etc

But in the database, the query will look like this,

SELECT *
FROM payments
WHERE created_time <= '2020-05-16 03:15:06' // created timestamp
AND id < '2a1aa856-ad26-4760-9bd9-b2fe1c1ca5aa' // this is UUID
ORDER BY created_time DESC
LIMIT 2

Benchmark Result

The drawback of this pagination method

The performance may not be the best like using the autoincrement id. But it’s consistent even we will have millions of data
Quite tricky and advanced, we need to understand the index, because if we didn’t add an index, this query will really take time on a big dataset. And also we need to careful when handling timestamps. Even I, still facing some issues when querying the timestamp when doing this.

The benefits of this pagination method

The ID is UUID, so it’s practically globally unique across microservice in the organizations.
The performance is consistent from the beginning until querying the last page of the data

Conclusions

Alright, after doing all the benchmark, I’ve come with some conclusions.

1. Performances: Faster to Slower

From the benchmark results (using the Golang benchmark tool), the faster one is using the autoincrement PK. See the chart below, the smaller the faster, the chart for the average-time needed for each operation in nanoseconds. This chart is might not be a good representation, it’s should be better if I make it in 95th, 97th, etc percentile, but I got this value from the benchmark result. So I assume this is already good enough for the representation.

https://medium.com/media/72d7697bf386da3132095d3d5983bbf8/href

Pagination with autoincrement ID is the faster, followed by UUID/created time, and PageNumber and LimitOffset. And this is only with 100K rows of data. And it will grow bigger as the data grow as well. So with only 100K data, even it still under 1 second, but the differences already quite high when using autoincrement compared to limit offset.

2. Development: Faster to Slower

Implementation difficulties from easy to hard

Using Offset , because we just only passing the offset and limit directly to the database.
Using PageNumber , this is opinionated, some people may have different logic, but for my case I put this in the top two.
Using autoincrement ID
Using UUID with created time

Code artifacts

For the code, I’ve pushed it to my GitHub repository, can be found here, https://github.com/bxcodec/go-postgres-pagination-example

The issues that I face when doing this

When doing all of these things, obviously I face some issues, but I’ve resolved it, and I also learn about this. I’ve written the things that I learned here, as well in this article: TIL: Becareful on Postgres Query, for Less than Or Equal on Timestamp

Author Suggestion

As a software engineer, and as the author of this article, I recommend to use autoincrement ID when doing pagination, but if your system or you don’t want to use autoincrement ID as the PK, you may consider of using keyset pagination, with my case using UUID + created_time timestamp.

Reference

Tons of Stackoverflow answer, I forgot which one, but all answers that related to pagination with Postgres.
Faster SQL Pagination with jOOQ Using the Seek Method
REST API Design: Filtering, Sorting, and Pagination https://medium.com/media/1f83e4a733ad3206f47e6dd38aa4fc6d/href

TIL: Becareful on Postgres Query, for Less than Or Equal on Timestamp

Iman Tumorang — Tue, 26 May 2020 04:21:09 +0000

Photo by Jonas Jacobsson on Unsplash

So a week ago, I do some experiments related to pagination on Postgres with combined UUID and timestamp. I got an interesting problem here, it’s happened when I’m doing a query for my created timestamp on the same timestamp.

Context

To give some context, I have a database schema like this.

And out of curiosity, I’m doing a load testing on my application, with many concurrent users up to 100 concurrent inserts. And this caused my database to have a lot of rows that have the same timestamp. For example can be seen below.

As you can see, the created_time was the same for all the records (may up to 100 records).

Problems

So, when I’m doing query based on the created time, there’s weird behavior, I thought it was a bug or something.

SELECT \* FROM payment\_with\_uuid 
WHERE 
created\_time **<= '2020-05-24 21:27:10'**
ORDER BY created\_time DESC LIMIT 10;

So, from that query, what I want to achieve is, I want to select all record that has the created_time less than or equal to (<=)the given timestamp.

But what I got instead is only all the timestamp that less than that created timestamp that is filtered. The timestamp that has the same value is not filtered.

So from this example rows below,

The result that I got is only Kane but Allistair and James is not filtered.

Another weird thing is if I reverse the query, like using greater than equal to (>=) syntax, I can fetch the others like Allistair and James but Kane will be out of order obviously.

Solutions and Things that I just Learned

The timestamp is an integer UNIX-timestamp under the hood

So, after searching on the whole internet, and Stackoverflow obviously, even asking my friend, I got the answer, that basically, the timestamp that stored in the database is UNIX-timestamp under the hood.

For example, if you see the database records from the above example, the stored timestamp is like this.

**2020-05-24 21:27:10**

Actually this is not the real value, because Postgres actually stores the UNIX-timestamp version of that rows. It may have nano or microseconds, so it’s not only stopped on the seconds version.

It may look like this,

**2020-05-24 21:27:10.37091**

We don’t know. Because for the sake of formatting, Postgres round it only to the second value.

So, I change my query to more specific, including the nano or microseconds, into looks like this, and which is works.

SELECT \* FROM payment\_with\_uuid 
WHERE 
created\_time **<= '2020-05-24 21:27:10.37091****'**
ORDER BY created\_time DESC LIMIT 10;

But then, there’s a question, how to do this from the application level. But for my case, since I’m using Golang, it’s quite easy.

Using RFC3339Nano On Golang

Since my application is built on top of Golang, to handle including the precision, I use time.RFC3339Nano from the time package in Golang. Its format looks like this, "2006–01–02T15:04:05.999999999Z07:00". Or you can see the details here

time format in Golang package

I don’t know how to do it in other programming languages, but the key is to include the precision as well for the query on the timestamp.

So in the case of doing a query that accepting the timestamp from the end-user, we need to format it first using the RFC3339Nano, and send it to the database. So the query still valid.

But why I need to use format, if I can use it without format? Because if we don’t format it, the time data struct will contain the milliseconds and nanoseconds as well? So to give the context, why I need this, especially in my case.

So I will use the created_time for pagination to fetch the next page as a cursor, I will get that timestamp, and convert it to a string. And that string will be used by the user for fetching a new page as a cursor.

And in my application, I will convert that string back to timestamp and use it for the query. And because of that, we need to do it carefully. Since in Golang, when we convert the timestamp to a string, it might not include the precision that we got from the Postgres. That’s why I need to format it with time.RFC3339Nano.

So I still keep maintaining the precision even I convert it to string and revert it back to timestamp.

References

Postgres: using timestamps for pagination — Stackoverflow (on the comment’s thread) https://medium.com/media/1f83e4a733ad3206f47e6dd38aa4fc6d/href * * *

Consuming SQS Message using Golang in EKS (Elastic Kubernetes Service) from AWS

Iman Tumorang — Mon, 09 Mar 2020 05:55:42 +0000

Photo by davide ragusa on Unsplash

A few days ago, I was working on SQS in Golang. Quite tricky but stressing enough for me. Because I stuck for 5 days only to make my consumer worked well in EKS.

To give some context, SQS stands for Simple Queue Service. It is a message queue service provided by AWS. More details about it, you can see on the official page of SQS from AWS.

To make it short, assume that I have an application that will need to consume a lot of messages from SQS.

Simply, we can use the AWS SDK Go for consuming the SQS message. But the flow is quite different from what I have experienced with Google Pubsub. In GCP, they’ve prepared a complete SDK that already has a function to long consuming/streaming the message from the Google Pubsub. In AWS, especially SQS, we need to make a long looping and making a REST call to pull the queued message.

So what we do is basically just copying this awesome article does “SQS Consumer Design: Achieving High Scalability while managing concurrency in Go”.

func (c \*consumer) Consume() {
 for w := 1; w <= c.workerPool; w++ {
  go c.worker(w)
 }
}func (c \*consumer) worker(id int) {
 for {
  output, err := retrieveSQSMessages(c.QueueURL, maxMessages)
  if err != nil {
   continue
  } var wg sync.WaitGroup
  for \_, message := range output.Messages {
   wg.Add(1)
   go func(m \*message) {
     defer wg.Done()
     if err := h(m); err != nil {
       //log error
       continue
     }
     c.delete(m) //MESSAGE CONSUMED
   }(newMessage(m))

   wg.Wait()
  }
 }
}

Yeap, I just copying his code. Because it’s looking good already. With worker pattern. So why just not using it right?

TLDR; But now, I have 2 problems

Chain Credentials Issue
Timeout Error when Consuming Message from SQS

Problem Statement

Chain Credentials

Our problem is not with the worker or stuff. It begins with us following the SDK introduction.

So the thing is, the default SDK has a sequential order for authentication. If you go to this page, “Configuring the AWS SDK for Go”, you will found about chain credentials order.

Chain Credentials AWS

From that documentation, we can learn that the SDK will look for ENV keys first. If not exists, then it will be looking to the shared credential file. And if the credential file also doesn’t exist, it will look at the IAM Role on EC2 and so on.

So, what’s the problem here is, in our use-case, since the default configurations of the SDK will load ENV variable first, if not exists, then it will be looking to the shared credentials file, it can be like AWS_WEB_IDENTITY_TOKEN_FILE. And for local workspace, it will look up to credentials file in AWS config which is located in ~/.aws/credentials (Mac/Linux) (for more info, you can read here).

It becomes a problem for us. Since the project will not be used only for us but maybe other engineers in a different team. We afraid that, to other engineers that have a configured AWS config in their locals, which is located in ~/.aws/credentials. We assume there will be a chance to them to run the application in local, but using production-access credentials that were configured in ~/.aws/credentials.

So, what we really want are:

Local development, we use ENV variable
On Staging and Production, we use IAM Role.
Avoid using the Shared Credentials File on the first place, to avoid accidentally using a production-access credentials file by the other engineers.

First Attempt Solution

Our first solution is, we will detect the IAM Role first if it exists then ok. If doesn’t exist, then our application will lookup for the ENV key which is AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

So then, we customized our chain credentials, into like this below

If you see the above functions, especially the credProviders , we specify the orders for the credentials chaining provider, first, it will look up to the Instance IAM Role, and then lookup for the ENV provider. So we basically remove the authentication using the shared credentials file. So whenever the engineers have a configured AWS credentials key in their PC which is located in ~/.aws/credentials , it still safe, since the SDK will look for the IAM Role and ENV key only.

The Issue

After making a custom chain credentials, we found another problem. It can be run well in EC2 instance, but not in our EKS. In short, our chain credentials only working on EC2 instances level. And for EKS case, it will work if we only allow IAM Role on the Nodes level.

For security-wise, for EKS, we want to use the IAM role on the service account level or in pods level, not at node level as we can find in the example here, “Introducing fine-grained IAM roles for service accounts”.

So it obvious, our custom chain credentials are not working well for EKS. So we need to change the chain method.

Final Solutions

To fix it, we then sit together with our infra engineers. Discuss and debugging the application. Takes a whole day, with many attempts, we finally decide to customized the solution logically in our application.

Turns out, to make it work in EKS, we have to enable the shared credentials file. It’s because in EKS we will use AWS_WEB_IDENTITY_TOKEN_FILE , based on this article.

So to use the web identity token file, we have to enable credentials using a file. Which are we avoid in the first place, but for the sake of work well in EKS, we then decide to enable chain auth with the shared credential file.

But to avoid engineer’s local shared credential file ~/.aws/credentials being used accidentally, we decide to handle it logically in the application (statically written in code).

We use an ENV variable APP_ENV to check if the environment is local or not.

// IsLocal will return true if the APP\_ENV is not listed in those
// three condition

func IsLocal() bool {
  envLevel := MustHaveEnv("APP\_ENV")
  return envLevel != "production" && 
         envLevel != "staging" &&   
         envLevel != "integration"
}

So no matter the APP_ENV’s value, if it’s not production or staging or integration we assume it as local.

As you can see, if the environment is not local (if !IsLocal()), we use the shared config file. And if it’s in local, we use ENV key, and it’s required, so we can avoid accidentally using production-access credentials in engineer’s local workspace ~/.aws/credentials.

With this, we finally solve our problem with credentials for the SDK safely. We will enforce the engineer’s to use ENV key in local (with using localstack). And will using shared credentials file in staging and production

Using Customized HTTP Client

Another problem, it’s when we trying to consume the SQS message. But before telling the details about that, to give some context, in the AWS SDK Documentation pages, we found this article, “Creating a Custom HTTP Client”. On that page, we found how to customize our HTTP Client.

Custom HTTP Client on AWS SDK

So then, we just follow these steps. We customized our HTTP Client. We set the timeout, the maximum idle connection, etc. And then, we run locally (thanks to localstack). We test it locally, and everything is fine. Perfectly worked.

The Issue

But then, when we tried to deploy it in our EKS. After deployed to EKS, we got a lot of errors.

time="2020-02-06T07:23:02Z" level=error msg="there was an error reading messages from SQS RequestError: send request failed\ncaused by: Post [https://sqs.ap-southeast-2.amazonaws.com/](https://sqs.ap-southeast-2.amazonaws.com/): net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2020-02-06T07:23:02Z" level=error msg="there was an error reading messages from SQS RequestError: send request failed\ncaused by: Post [https://sqs.ap-southeast-2.amazonaws.com/](https://sqs.ap-southeast-2.amazonaws.com/): net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

Actually this is not really a big problem, but it will annoying and increase our log storage because this will run realtime due to our long-running pulling message from SQS using HTTP call. Solving this, I take 2 days!!! My whole 2 days ruined because of this. WTF!

Final Solutions

I was frustrated, there’s no one who understands why this issue happening in my team. And then, I decided to ask in Gopher’s slack group.

Thanks to André Eriksson and Zach Easey and others in Slack Gopher https://gophers.slack.com/archives/C029RQSEE/p1580979593499800. They help me to solve my problem here.

So to summarize, this has happened because we use long polling enabled, but with the long polling, connection enabled, we also customized our HTTP client.

So, the solution is, for long polling consume the message from SQS, we shouldn't customize our HTTP client, we use the default one. Because, if we customized the HTTP client (like the timeout, etc) it will compete with the long polling connection time, and that’s causing the request canceled (Client.Timeout exceeded while awaiting headers) errors happened a lot.

This is quite tricky, because as far as I know, to avoid the bad timeout and user experience, I usually customized my HTTP Client. But for this case, for SQS, since we enable the long polling call, the right solution is only using the default HTTP Client that doesn’t have the timeout settings.

Conclusions

The takeaways from this whole projects when integrating SQS with Go in EKS, are:

We manually handle the custom chain credentials for the AWS SDK to avoid accidentally using production-access credentials file in the local workspace engineers, ~/.aws/credentials.
To enable the IAM role on the pods level, we need to allow the SDK to use Shared Credentials file, which is quite tricky since we disable it on the local workspace.
We handle the chain credentials logically in code, using the extra ENV key, like APP_ENV.
If you enable the long polling connection to consume the SQS message, we must use default HTTP client, don’t customize its timeout, or you will face a lot of timeout error. https://medium.com/media/1f83e4a733ad3206f47e6dd38aa4fc6d/href

HTTP — PATCH Method! I’ve Thought the Wrong Way!!!

Iman Tumorang — Tue, 18 Feb 2020 09:40:29 +0000

Photo by Free To Use Sounds on Unsplash

I’ve been working on Software Engineering for more than 3 years now. And I’m surprised, in these 3 years, I never face any PATCH endpoint yet, until now.

And because of that, I never really use the PATCH Method before, and never develop it either. I remember 3 years ago when the first time I asked my senior about the difference between PUT and PATCH was only the changed value. PUT will replace the entire item, and PATCH changing the specified field.

That’s it! But in reality, in my careers, we never used the PATCH method. The reason is that we still didn’t need it at that time. So I now live with this belief, and whenever I talk to others, they usually understand what I mean.

My Original Thought about PATCH

So, since I know about HTTP Method and REST. I only use these methods, GET, DELETE, POST, and PUT. I never have a chance (yet) to use PATCH.

We know clearly, how the PUT work. And I’ve been using PUT for a long time now. For example, I have this JSON. And I’m doing PUT operation.

GET /user/bxcodec
{
  "name": "Iman",
  "username": "bxcodec"
}

Applying HTTP PUT

PUT /user/bxcodec
{
  "name": "Iman Tumorang",
  "username": "bxcodec"
}

Final Results

GET /user/bxcodec
{
  "name": "Iman Tumorang",
  "username": "bxcodec"
}

And then, derived from PUT, then PATCH, from what people say, “ PATCH not replacing the entire item, but only the specified one ”

So, from that statement, in default, this is what I think how PATCH work. For example, I have this JSON.

GET /user/bxcodec
{
  "name": "Iman",
  "username": "bxcodec"
}

Applying HTTP Patch

PATCH /user/bxcodec
{
  "name": "Iman Tumorang"
}

Final Results

GET /user/bxcodec
{
  "name": "Iman Tumorang",
  "username": "bxcodec"
}

Right? This is how I think HTTP-PATCH should work. And I live with this belief for more than 3 years.

The Plot Twists

But, just now, when I write this, I found this on Golang docs here: https://golang.org/pkg/net/http/

Out of curiosity, I notice a different value there, I found a comment for RFC 5789. Why the heck it has a different RFC, not like the others? Why is it different?

After looking at the RFC 5789, I found a weird thing. From the description,

RFC 5789

With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version.

So, from the RFC docs, it says, in the request body, I have to specify a set of actions or instructions to modify the current resource in request URI. As we can see in the example above. But how I do set the actions in the request body? How should I format it?

Introducing JSON PATCH

To answer the previous question, how should I format the action list in my request body? I found an interesting thing, it’s another RFC called JSON-Patch (RFC 6902). This is an RFC to explain how to use JSON-Patch for JSON operations.

A short example of how JSON-PATCH worked is more like this. For example, we have a JSON here.

{
  "name": "Iman",
  "username": "bxcodec"
}

Using the JSON-PATCH.

[
  { "op": "replace", "path": "/name", "value": "Iman Tumorang" },
  { "op": "add", "path": "/likes", "value": ["go", "blogging"] }
]

Patched results.

{
  "name": "Iman Tumorang",
  "username": "bxcodec",
  "likes": [
    "go",
    "blogging"
  ]
}

So in JSON-PATCH, I have to define the operations, the path, and the value for modifying the resources. And all the available commands can be seen in the RFC6902.

So, here the plot twist, surprisingly, HTTP PATCH and JSON-PATCH have a correlation. When I open the RFC 6902, I got this.

RFC 6902

JSON Patch is a format (identified by the media type “application/ json-patch+json”) for expressing a sequence of operations to apply to a target JSON document; it is suitable for use with the HTTP PATCH method.

Gotcha!

So, in simple, for HTTP-PATCH, we can use JSON-PATCH for the request body. So the action list for the request body, we can define it using JSON-PATCH.

So, to combine HTTP-PATCH and JSON-PATCH, if I put it into an example, it will be more like this below.

GET /user/bxcodec
{
  "name": "Iman",
  "username": "bxcodec"
}

Applying HTTP Patch

PATCH /user/bxcodec
[
  { "op": "replace", "path": "/name", "value": "Iman Tumorang" }
]

Final Results.

GET /user/bxcodec
{
  "name": "Iman Tumorang",
  "username": "bxcodec"
}

Conclusions

Today, after living with a wrong belief, I learned new things. HTTP-PATCH has a different way to implement it. We can use JSON-PATCH as the request-body on the HTTP-PATCH request method.

But, I don’t know, to be honest, I still feel this way is not really convenient. And it adds an extra layer of our implementations, compares to like what I used to be thought. Luckily, there’s already a lot of JSON-PATCH library out there, like https://github.com/evanphx/json-patch for Golang, https://www.npmjs.com/package/fast-json-patch in Typescript, and many more. So I don’t think it will be hard to implement it in my next projects. But yeah, let’s see haha.

But yeah, even it more convenient to change directly the value for patching (without using JSON-PATCH, and not following the RFC 5789) LOL, I will try to follow this in my next projects.

https://medium.com/media/1f83e4a733ad3206f47e6dd38aa4fc6d/href

One of the Important Skills that Every Engineer Should Have: Write A good README

Iman Tumorang — Tue, 29 Oct 2019 06:13:42 +0000

Photo by Nour Wageh on Unsplash

As nowadays, Software is really shaping the whole world. There are so many companies and products built by technology that help the human race in their activity. Let’s say, when I want to discuss with someone, I just open WhatsApp or Telegram, and I can ask all of my friends there.

When I want to book a hotel or just a small room, there’s already AirBnB. Flight ticket? There are Traveloka or Tiket, etc. For Transportations? We can use Gojek or Grab. Hungry? Go-Food or Grab-Food. (I’m talking about tech-product in SEA especially in Indonesia).

All of those products were created by the technology itself. And no wonder, Software Engineer becomes the hottest job nowadays. To make a great product, it needs great software. And to make the great software, need the great engineers behind them too.

Not only commercial products but open-source products also become going popular now. Let’s say like Kubernetes, Docker, Golang, NodeJS, etc.

Not only the big open-source project. There are a lot of small open-source projects too, like a library for specific functions. And it can be written in Golang, NodeJS, Java, etc.

But, the problem is, there’s a lot of projects that might be useful to society but it’s doesn’t have a good or descriptive README. And it will make the library or open-source project become useless or my uncle said: “not classy”.

If this happens for the library (doesn’t have the README), then we can just switch to another library that has a similar function and has a better README, to explain how to use it, how to run and install any dependencies.

But what happens if this happens to internal projects? For the internal company's project? Can we switch to another service that has a similar function? LOL! 🔥

What is README?

Anyway, before moving further, let’s find what’s the meaning of README? Why is it important? How to create it?

So, if we look from Wikipedia,

A README file contains information about other files in a directory or archive of computer software. A form of documentation, it is usually a simple plain text file called READ.ME, README.TXT,[1] README.md (for a text file using markdown markup), README.1ST — or simply README ~ https://en.wikipedia.org/wiki/README

In general, README is just a simple general information of software/project that exists in the same directory/repository of the project. README is used to explain the details project in a short way. Like, what is the project used for? What are the dependencies, how to install it, how to compile it, or how to run it in local? Where to go when there’s an issue. What is the current status of the project, is it experimental, production, or just still on development.

We put all of that information in the README. So when another engineer reads that project, they can understand the project just in 5 minutes without having to look to the source code, what is the project used for, like the function or the endpoint (if REST API projects), etc.

Why is it important?

Yeah, why this is so important?

Before saying this is important, I want to clarify first, all content I write here is purely in my opinion. You can be agreed or not, but I’m saying what is the right thing based on my experience. If you do not agree, let’s discuss in the comments. Maybe I’m wrong, maybe I've experienced the wrong practice because I also want to learn what’s the right one.

So why this is important?

First, let me ask a question, what is a good software? How do we define that a software is good enough?

In my opinion, one of the parameters to say software is “Good” is when that software is useful to others. Meaning that it’s used by others not only us who developed it but other people too. It can be engineers, it can be users, it can be a system, it can anything that used that software.

And building software is really cost us much time, and maybe money so it can be used by others. Also, when building that software, we maybe working on the team. Writing the same product to make useful software.

And that’s when we have to think about README projects. I have a few reasons why we need README in our projects. Such as:

Engineer Movement
Visibility and Development Pace
We’re not the only one!

Engineers Movement

Photo by Adri Tormo on Unsplash

When working in a company, it’s true that we won’t work on the same project in our lifetime. There will be a new interesting project or maybe we will move to another team. Or maybe we will be promoted to a higher level, so we won’t be doing coding again.

If this happens, the new engineer that will responsible for our sh*t will be having a hard time for the transitions. It’s okay if it’s only one repository. But what happens if we leave many repositories that we maintained. The new engineer that replacing our positions will be having a hard time and stress.

Or maybe, we worked on a team. And all of our team members already understand very clear the whole projects including other project repositories. And even without the README, every team member already understands all of the repositories used for. And then we started to think, README is not really important. Because everyone already understands all of the repositories that our team maintained.

But, the things is, if our company grows and scaled, so eventually, our team will be growing too, we will hire a new engineer to help our team to manage the services. Unless the team is super-engineers or we call 10X engineers, that can maintain everything only by just a few engineers without hiring new engineers, if that’s the case, my apologies to underestimate the team’s skill. 🙇‍♂️

So, no matter how boring is writing the README, please keep in mind, we’re not the only one who will be working on the project. We also need to think about the future hired engineer that will handle all of these projects.

Let me tell you a dark story, I’ve experience working on a project that doesn’t have a README. It takes 3 weeks for me only to make it run in my local. There’s no guidance. I need to look to source code, what are the configurations, such as database URI, PubSub env variable, etc. And it’s really frustrating!

Visibility and Development Pace

Photo by Marc-Olivier Jodoin on Unsplash

Only by seeing the README, we can understand what is the repository used for, without spending times to look on the source code.

Let me give you a comparison. I have 2 repositories here:

In 5 seconds, can you understand how to use both of them in your project? Both of that repositories are a library for math operations for multiplication. Which one is more understandable?

So that’s one of the reasons why we need to put README in our projects. It doesn’t have to fancy, has color and everything. At least, we can use it without asking the person who built it. We can import it to our projects (if it’s a library). Or we can run it locally if it a service or REST API project.

And the faster engineers to understand about the repositories, the faster they will work. And the development pace will still the same from the start to the end.

We’re not the only one!

Photo by Austin Distel on Unsplash

Then there’s a question, “what if I built software for myself? It’s useful for me, but maybe not useful to others”. Well, if it just for personal consumption. It’s up to you to create the README or not. But, when you solve a problem for yourself, it can be people’s problem too. So, putting a good readme maybe help others to use it.

Let me tell another story that happened a few months ago, it’s when I’m developing a feature in my side projects. I found a blog post of an engineer that says he creates a library as his side project just for fun. But I think, what he worked on is amazing. He creates an Angular plugin for the text editor, which has the same function as Medium has.

But when I’m looking to the repository in Github. (I don’t want to post it here, because I don’t want to make it look bad since he built it for fun). I read the README, and it is not updated. I can’t use the library. WTF. So then I just passed it and looking for other libraries that have similar functions that have better README and guidance on how to use one.

So, what I’m saying is even it’s just for a fun project. Not used in production or in our company. Putting an updated and descriptive README will help millions of people out there. We don’t know that what we worked for side projects will useful to others on another side of the earth.

Takeaways

So takeaways from this story are we have to create the README in every project we worked on.

Even I still learning and enforce myself to create README in all of my projects. I remember in my previous company, sometimes I also too lazy to put the README. But at the time I don’t think about how if the team is scaled, or I don’t have many repositories and projects to maintain, so I never consider the README in my past experience.

So to be honest, I also just learning the importance of README, after I finally experience the hard time to learn application that doesn’t have a README. I’m totally confused and stressed. And it’s not fun! NOT FUN!

https://medium.com/media/1f83e4a733ad3206f47e6dd38aa4fc6d/href

Understanding about RFC 3339 for Datetime Formatting in Software Engineering

Iman Tumorang — Mon, 14 Oct 2019 13:34:26 +0000

Small and simple things that not every engineer knows, but very important things that every engineer should understand.

Photo by Aron Visuals on Unsplash

A few days ago, when building the Mabar, we have a great debate about Datetime formatting. There’s an issue that our timezone is not consistent. So, later after a long fight and discussion, we come to an agreement to use a standard for our timezone.

We finally decide to use the RFC 3339 as the standard for the date-time format. This means, both backend and frontend will use this format to communicate about the DateTime format. And also, we agreed to use UTC+0 as the default timezone, even when creating the event and receiving the event detail from the server. And let both frontend and backend convert it based on their timezone.

So what is RFC 3339?

Introduction RFC 3339

RFC stands for Request For Comment. RFC is a formal document from the Internet Engineering Task Force (IETF) that is the result of committee drafting and subsequent review by interested parties.

There are already so many RFC documents released by this committee. And become standard in every business. One of their documents is the RFC 3339, a document for DateTime formatting. The link for the RFC 3339 can be accessed here: https://www.ietf.org/rfc/rfc3339.txt

Generally, if you look at the RFC document it mainly discusses the DateTime formatting, and to summarize you will see how the proposed DateTime format like the example below.

2019-10-12T07:20:50.52Z

Yap, just like that.

But there’s a question. For Datetime formatting, there’s already standardized by the ISO. It’s ISO 8601 standard for Datetime. And for people that already familiar with ISO 8601, the RFC 3339 is pretty similar. What’s different?

So, if you look again at the RFC document. RFC 3339 use/follow the ISO 8601 profile for the Internet DateTime. Clearly stated in chapter 5.6

RFC 3339 Chapter 5.6

So actually, there are no big differences between Date ISO 8601. The only small thing that makes these 2 different is the “T” syntax between date and time. ISO 8601 uses the “T” character to separate the date and time. In RFC 3339, you can replace the “T” character with only using space.

For example:

# This is acceptable in ISO 8601 and RFC 3339 (with T)

2019-10-12T07:20:50.52Z

# This is only accepted in RFC 3339 (without T)
2019-10-12 07:20:50.52Z

Just it. So overall, it still the same as the ISO 8601.

Understanding about Time Zone

I’m a bit shocked when I’m asking about RFC 3339 to around my close friend, there’s no one understand about this. Even for ISO 8601, it’s only a few of them who know the details about this.

Especially reading the TimeZone format. From 1–10 of my friend, I can say only 1 understand about the timezone format.

2019-10-12T07:20:50.52Z

Take a look above example. When I asked someone about it, they will say it is 2019–10–12 07:20:50.52 in Jakarta time, because we live in Jakarta. So he will assume it was Jakarta time. But that’s wrong. Because if we convert that time to Jakarta time it should be: 2019–10–12 14:20:50.52.

How is it possible? How do I can say that the Jakarta time of that example is should be 2019–10–12 14:20:50.52?

In RFC 3339, we can also know the time-zone from the format. It displayed in the “ Z” syntax. “Z” means UTC+0. “Z” stands for Zulu timezone which is the same with GMT or UTC (https://stackoverflow.com/a/9706777/4075313). So if we put Z on the DateTime, it’s mean its timezone is UTC+0.

More detailed example:

2019-10-12T07:20:50.52Z  #(UTC+0)
2019-10-12T07:20:50.52+00:00 #(UTC+0)
2019-10-12T14:20:50.52+07:00 #(UTC+7)
2019-10-12T03:20:50.52-04:00 #(UTC-4)

Look at the bold text. It will explain how the timezone is written in the DateTime format.

Why Is It Important?

This is really important to handle the request that comes from many timezones. If your application only accepts requests as the same as your timezone (for now), then this might be not really treated well by the engineer.

And for the engineer, why is it really important even a few programming languages like Golang already handle this? It is because sometimes the engineer not understand it very well, and may come brings a new issue later.

Let me tell you a story.

So, I will tell an example of why this is really important. This has just happened recently to me and my friend, not in Mabar, but my friend from my office. We have an issue about transactions pending that we need to take a look at and verified.

In our application, we use UTC+0, and he(my friend) knows this. So to fix the issue, I just need to get the details and check if there’s another transaction that pending around the pending one.

So, the problem comes are when I asked about the transaction time (created time) of the row in Database. He adds the hour automatically with our timezone which is UTC+7, but not change the timezone format. And it brings an extra time for me to verify all transactions of that time that should be intended to. Because I’m querying the wrong DateTime range.

For example,

#This is our DateTime stored in DB
2019-10-12T07:20:50.52Z

Then I asked him to tell me the transaction time through chat. And he gives this to me.

# He adds the hour with +7 but not change the timezone

2019-10-12T14:20:50.52Z

So what’s wrong here? Look at the “hour” value. He added it with 7 and send it to me. But the timezone is still “Z” (UTC+0). Since I know to read the RFC 3339, I’m thinking this is UTC+0 because he wrote it “Z”.

And, with that date-time, I’m doing some queries to DB. But, what I got is not exactly what I want. And when I’m trying to query by the ID, it shows different than what he gives to me. I’m afraid there’s an update query to DB that causing it different than what he gives to me.

And it brings confusion to me. And to verify it, I ask him again, is it UTC+0 value or not, and he said, that he already added +7 to the hour. And I feel like arghhh…. 🤯

And then, I say to him, when adding the hour, please change the Timezone too. So it does not bring confusion to whoever will read the date-time.

Takeaways

This is a really important thing for Engineers to understand the format to avoid inconsistent data. Luckily, from my experience above, it stills me who read the given date-time. What if there’s an application that will read the Datetime, then there will be so much inconsistency data, because we add manually and it’s the wrong timezone.

So here are the takeaways:

“Z”: stands for Zero timezone (UTC+0). Or equal to +00:00 in the RFC 3339.
RFC 3339 is following the ISO 8601 DateTime format. The only difference is RFC allows us to replace “T” with “space”.

References:

https://www.ietf.org/rfc/rfc3339.txt https://medium.com/media/1f83e4a733ad3206f47e6dd38aa4fc6d/href

Today I Learned — Fix: go get private repository return error reading sum.golang.org/lookup

Iman Tumorang — Mon, 23 Sep 2019 04:02:25 +0000

Fixing error: go get gomodule on private module with error message pattern verifying git-host… reading sum.golang.org/lookup … 410 gone

Today on a bright day, Sunday Funday, but ruined by something that made me furious for an hour. Actually, I’ve stuck with this issue since last night (Saturday Night), but since I’m really tired, I decide to stop and rest and continue again on Sunday.

So, I’ve been working on Mabar (https://mabar.id) as one of my side projects. It’s still on the beta version, still lack of many features, you can try it on Android Play Store here

I have a high expectation on this Mabar, it should be a platform, that available on many interfaces (Mobile, Desktop, Web). For the tech-stack, I use Golang as the backend, Kubernetes as the infrastructure, and Digital Ocean behind them as the Server.

Problems

So the issues is, I have a private module (Golang module), a simple library that will be imported by my backend API. But somehow, I can’t get the module and always bring errors when I do the go get command.

Let’s say the package name is lucifer. And it always throw error on the terminal. That really made me mad on it.

$ go get -v bitbucket.org/compay/lucifer
go: finding bitbucket.org/compay/lucifer latest
go: downloading bitbucket.org/compay/lucifer v0.0.0-20190921175342-61a76c096369
**verifying bitbucket.org/compay/lucifer@v0.0.0-20190921175342-61a76c096369: bitbucket.org/compay/lucifer@v0.0.0-20190921175342-61a76c096369: reading** [**https://sum.golang.org/lookup/bitbucket.org/**](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369) **compay** [**/lucifer@v0.0.0-20190921175342-61a76c096369**](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369) **: 410 Gone**

If you look the message, it says the package is gone or not available in sum.golang.org.

At first, I think this happens because I forgot to enforce SSH on bitbucket, as I ever write it here: https://medium.com/easyread/today-i-learned-fix-go-get-private-repository-return-error-terminal-prompts-disabled-8c5549d89045

But, it still not worked well. It still keeps returning an error when I do the go get commands, even I force it only using SSH.

Root Causes

So, after searching for this issue on the internet, I found the root-causes. This only happens on Golang from version 1.13. I can verify this after reading this release https://golang.org/doc/go1.13#modules

So, this happened because there’s a new feature about proxy in this Golang version.

Solutions

Actually there are a few solutions that we can choose.

Using GOPRIVATE

As stated in the release doc of Go 1.13,

The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.

Means, to solve the issue above, we can just fill the GOPRIVATE variable in our system. Add this command in the ~/.bashrc. *Change the export value based on your company/org name.

**export**  **GOPRIVATE** ="gitlab.com/ **idmabar** ,bitbucket.org/ **idmabar** ,github.com/ **idmabar**"

And to verify if this worked, you can do the go env command. It’s should be more like this.

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/imantumorang/Library/Caches/go-build"
GOENV="/Users/imantumorang/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"  
GOOS="darwin"
GOPATH="/Users/imantumorang/go"
**GOPRIVATE="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"**
GOPROXY="[https://proxy.golang.org,direct](https://proxy.golang.org,direct)"
GOROOT="/usr/local/Cellar/go/1.13/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13/libexec/pkg/tool/darwin\_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO\_ENABLED="1"
GOMOD=""
CGO\_CFLAGS="-g -O2"
CGO\_CPPFLAGS=""
CGO\_CXXFLAGS="-g -O2"
CGO\_FFLAGS="-g -O2"
CGO\_LDFLAGS="-g -O2"

And now I can do the go get command for my private repository.

$ go get bitbucket.org/company/lucifer
go: finding bitbucket.org/company/lucifer latest
go: downloading bitbucket.org/company/lucifer v0.0.0-20190921175342-61a76c096369
go: extracting bitbucket.org/company/lucifer v0.0.0-20190921175342-61a76c096369

So this env-variable will tell the go get command to use the private host proxy to retrieve the package.

Using GONOSUMDB

Another solution, maybe using the GONOSUMDB variable. I still don’t try this, but it seems works after reading this proposal https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md

So you can set this in your environment variable:

**export**  **GONOSUMDB** ="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"

Actually this issue only happens in new Golang version 1.13 and afterward. So before updating your Golang version, make sure to set this environment variable.

Here a few links that may be related to this issue, thanks to noveaustack for finding this and posting this in Stackoverflow, I just reposted this because I just know this issue and as a new thing I learned.

References

Stackoverflow answer: https://stackoverflow.com/a/57887036/4075313
Proposal about Go Sum DB for go module: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Proxying Checksum DB: https://docs.gomods.io/configuration/sumdb/
Github Issues related to this issue: #33985 and #32291