DEV Community: Oliver Nguyen

iter.json: A Powerful and Efficient Way to Iterate and Manipulate JSON in Go

Oliver Nguyen — Fri, 13 Dec 2024 02:54:47 +0000

Have you ever needed to modify unstructured JSON data in Go? Maybe you’ve had to delete all blacklisted fields, rename keys from camelCase to snake_case, or convert all number ids to strings because JavaScript does not like int64? If your solution has been to unmarshal everything into a map[string]any using encoding/json and then marshal it back... well, let’s face it, that’s far from efficient!

What if you could loop through the JSON data, grab the path of each item, and decide exactly what to do with it on the fly?

Yes! I have a good news! With the new iterator feature in Go 1.23, there’s a better way to iterate and manipulate JSON. Meet ezpkg.io/iter.json — your powerful and efficient companion for working with JSON in Go.

1. Iterating JSON

Given that we have an alice.json file:

{
  "name": "Alice",
  "age": 24,
  "scores": [9, 10, 8],
  "address": {
    "city": "The Sun",
    "zip": 10101
  }
}

First, let's use for range Parse() to iterate over the JSON file, then print the path, key, token, and level of each item. See examples/01.iter.

package main

import (
    "fmt"

    "ezpkg.io/errorz"
    iterjson "ezpkg.io/iter.json"
)

func main() {
    data := `{"name": "Alice", "age": 24, "scores": [9, 10, 8], "address": {"city": "The Sun", "zip": 10101}}`

    // 🎄Example: iterate over json
    fmt.Printf("| %12v | %10v | %10v |%v|\n", "PATH", "KEY", "TOKEN", "LVL")
    fmt.Println("| ------------ | ---------- | ---------- | - |")
    for item, err := range iterjson.Parse([]byte(data)) {
        errorz.MustZ(err)

        fmt.Printf("| %12v | %10v | %10v | %v |\n", item.GetPathString(), item.Key, item.Token, item.Level)
    }
}

The code will output:

|         PATH |        KEY |      TOKEN |LVL|
| ------------ | ---------- | ---------- | - |
|              |            |          { | 0 |
|         name |     "name" |    "Alice" | 1 |
|          age |      "age" |         24 | 1 |
|       scores |   "scores" |          [ | 1 |
|     scores.0 |            |          9 | 2 |
|     scores.1 |            |         10 | 2 |
|     scores.2 |            |          8 | 2 |
|       scores |            |          ] | 1 |
|      address |  "address" |          { | 1 |
| address.city |     "city" |  "The Sun" | 2 |
|  address.zip |      "zip" |      10101 | 2 |
|      address |            |          } | 1 |
|              |            |          } | 0 |

2. Building JSON

Use Builder to build a JSON data. It accepts optional arguments for indentation. See examples/02.builder.

b := iterjson.NewBuilder("", "    ")
// open an object
b.Add("", iterjson.TokenObjectOpen)

// add a few fields
b.Add("name", "Alice")
b.Add("age", 22)
b.Add("email", "alice@example.com")
b.Add("phone", "(+84) 123-456-789")

// open an array
b.Add("languages", iterjson.TokenArrayOpen)
b.Add("", "English")
b.Add("", "Vietnamese")
b.Add("", iterjson.TokenArrayClose)
// close the array

// accept any type that can marshal to json
b.Add("address", Address{
    HouseNumber: 42,
    Street:      "Ly Thuong Kiet",
    City:        "Ha Noi",
    Country:     "Vietnam",
})

// accept []byte as raw json
b.Add("pets", []byte(`[{"type":"cat","name":"Kitty","age":2},{"type":"dog","name":"Yummy","age":3}]`))

// close the object
b.Add("", iterjson.TokenObjectClose)

out := errorz.Must(b.Bytes())
fmt.Printf("\n--- build json ---\n%s\n", out)

Which will output the JSON with indentation:

{
    "name": "Alice",
    "age": 22,
    "email": "alice@example.com",
    "phone": "(+84) 123-456-789",
    "languages": [
        "English",
        "Vietnamese"
    ],
    "address": {"house_number":42,"street":"Ly Thuong Kiet","city":"Ha Noi","country":"Vietnam"},
    "pets": [
        {
            "type": "cat",
            "name": "Kitty",
            "age": 2
        },
        {
            "type": "dog",
            "name": "Yummy",
            "age": 3
        }
    ]
}

3. Formatting JSON

You can reconstruct or format a JSON data by sending its key and values to a Builder. See examples/03.reformat.

{
    // 🐝Example: minify json
    b := iterjson.NewBuilder("", "")
    for item, err := range iterjson.Parse(data) {
        errorz.MustZ(err)
        b.AddRaw(item.Key, item.Token)
    }
    out := errorz.Must(b.Bytes())
    fmt.Printf("\n--- minify ---\n%s\n----------\n", out)
}
{
    // 🦋Example: format json
    b := iterjson.NewBuilder("👉   ", "\t")
    for item, err := range iterjson.Parse(data) {
        errorz.MustZ(err)
        b.AddRaw(item.Key, item.Token)
    }
    out := errorz.Must(b.Bytes())
    fmt.Printf("\n--- reformat ---\n%s\n----------\n", out)
}

The first example minifies the JSON while the second example formats it with prefix "👉" on each line.

--- minify ---
{"name":"Alice","age":24,"scores":[9,10,8],"address":{"city":"The Sun","zip":10101}}
----------

--- reformat ---
👉   {
👉       "name": "Alice",
👉       "age": 24,
👉       "scores": [
👉           9,
👉           10,
👉           8
👉       ],
👉       "address": {
👉           "city": "The Sun",
👉           "zip": 10101
👉       }
👉   }
----------

4. Adding line numbers

In this example, we add line numbers to the JSON output, by adding a b.WriteNewline() before the fmt.Fprintf() call. See examples/04.line_number.

// 🐞Example: print with line number
i := 0
b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)
    b.WriteNewline(item.Token.Type())

    // 👉 add line number
    fmt.Fprintf(b, "%3d    ", i)
    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- line number ---\n%s\n----------\n", out)

This will output:

  1    {
  2        "name": "Alice",
  3        "age": 24,
  4        "scores": [
  5            9,
  6            10,
  7            8
  8        ],
  9        "address": {
 10            "city": "The Sun",
 11            "zip": 10101
 12        }
 13    }

5. Adding comments

By putting a fmt.Fprintf(comment) between b.WriteComma() and b.WriteNewline(), you can add a comment to the end of each line. See examples/05.comment.

i, newlineIdx, maxIdx := 0, 0, 30
b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    b.WriteComma(item.Token.Type())

    // 👉 add comment
    if i > 0 {
        length := b.Len() - newlineIdx
        fmt.Fprint(b, strings.Repeat(" ", maxIdx-length))
        fmt.Fprintf(b, "// %2d", i)
    }
    i++

    b.WriteNewline(item.Token.Type())
    newlineIdx = b.Len() // save the newline index

    b.Add(item.Key, item.Token)
}
length := b.Len() - newlineIdx
fmt.Fprint(b, strings.Repeat(" ", maxIdx-length))
fmt.Fprintf(b, "// %2d", i)

out := errorz.Must(b.Bytes())
fmt.Printf("\n--- comment ---\n%s\n----------\n", out)

This will output:

{                             //  1
    "name": "Alice",          //  2
    "age": 24,                //  3
    "scores": [               //  4
        9,                    //  5
        10,                   //  6
        8                     //  7
    ],                        //  8
    "address": {              //  9
        "city": "The Sun",    // 10
        "zip": 10101          // 11
    }                         // 12
}                             // 13

6. Filtering JSON and extracting values

There are item.GetPathString() and item.GetRawPath() to get the path of the current item. You can use them to filter the JSON data. See examples/06.filter_print.

Example with item.GetPathString() and regexp:

fmt.Printf("\n--- filter: GetPathString() ---\n")
i := 0
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)

    path := item.GetPathString()
    switch {
    case path == "name",
        strings.Contains(path, "address"):
        // continue
    default:
        continue
    }

    // 👉 print with line number
    fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath())
}

Example with item.GetRawPath() and path.Match():

fmt.Printf("\n--- filter: GetRawPath() ---\n")
i := 0
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)

    path := item.GetRawPath()
    switch {
    case path.Match("name"),
        path.Contains("address"):
        // continue
    default:
        continue
    }

    // 👉 print with line number
    fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath())
}

Both examples will output:

 2              "Alice" . name
 9                    { . address
10            "The Sun" . address.city
11                10101 . address.zip
12                    } . address

7. Filtering JSON and returning a new JSON

By combining the Builder with the option SetSkipEmptyStructures(false) and the filtering logic, you can filter the JSON data and return a new JSON. See examples/07.filter_json

// 🦁Example: filter and output json
b := iterjson.NewBuilder("", "    ")
b.SetSkipEmptyStructures(true) // 👉 skip empty [] or {}
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    if item.Token.IsOpen() || item.Token.IsClose() {
        b.Add(item.Key, item.Token)
        continue
    }

    path := item.GetPathString()
    switch {
    case path == "name",
        strings.Contains(path, "address"):
        // continue
    default:
        continue
    }

    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- filter: output json ---\n%s\n----------\n", out)

This example will return a new JSON with only the filtered fields:

{
    "name": "Alice",
    "address": {
        "city": "The Sun",
        "zip": 10101
    }
}

8. Editing values

This is an example for editing values in a JSON data. Assume that we are using number ids for our API. The ids are too big and JavaScript can't handle them. We need to convert them to strings. See examples/08.number_id and order.json.

Iterate over the JSON data, find all _id fields and convert the number ids to strings:

b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    key, _ := item.GetRawPath().Last().ObjectKey()
    if strings.HasSuffix(key, "_id") {
        id, err0 := item.Token.GetInt()
        if err0 == nil {
            b.Add(item.Key, strconv.Itoa(id))
            continue
        }
    }
    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- convert number id ---\n%s\n----------\n", out)

This will add quotes to the number ids:

{
    "order_id": "12345678901234",
    "number": 12,
    "customer_id": "12345678905678",
    "items": [
        {
            "item_id": "12345678901042",
            "quantity": 1,
            "price": 123.45
        },
        {
            "item_id": "12345678901098",
            "quantity": 2,
            "price": 234.56
        }
    ]
}

Conclusion

The ezpkg.io/iter.json package empowers Go developers to handle JSON data with precision and efficiency. Whether you need to iterate through complex JSON structures, build new JSON objects dynamically, format or minify data, filter specific fields, or even transform values, iter.json offers a flexible and powerful solution.

I’m excited to share this package with the community as a tool for effective JSON manipulation without the need for fully parsing the data. While it’s still in early development and there’s room for more features, it already works well for many common use cases.

If you have specific requirements or ideas for improvement, feel free to reach out — I’d love to hear your feedback and help support your use cases! 🥳

Author

I'm Oliver Nguyen . A software engineer working with Go and JS. I enjoy learning and seeing a better version of myself each day. Occasionally spin off new open source projects. Share knowledge and thoughts during my journey.

The post is also published at olivernguyen.io.

Refactor: GoroutineTracker with unnecessary usage of reflect

Oliver Nguyen — Wed, 03 Jul 2024 16:00:00 +0000

Today, I encountered this code in my company codebase (the code and comments are rewritten for demo purpose and do not include any proprietary code):

type GoroutineTracker struct {
  wg sync.WaitGroup
  // ... some other fields
}
// Go starts a new goroutine and tracks it with some metrics.
func (g *GoroutineTracker) Go(ctx context.Context, name string, f any, args ...any) {
    fn := reflect.TypeOf(f)
    if fn.Kind() != reflect.Func { panic("must be function") }
    if fn.NumIn() != len(args) { panic("args does not match fn signature") }
    if fn.NumOut() > 0 { panic("output from fn is ignored") }

    g.wg.Add(1)
    id := g.startCaptureTime()
    go func() {
        defer func() {
            r := recover()
            // ... some panic handling code
            g.wg.Done()
            g.endCaptureTime(id)
        }()

        input := typez.MapFunc(args, func(arg any) reflect.Value {
            return reflect.ValueOf(arg)
        })
        _ = reflect.ValueOf(f).Call(input)
    }()
}
// Wait for all goroutines to finished.
func (g *GoroutineTracker) Wait() { g.wg.Wait() }

The GoroutineTracker is used for tracking usage of goroutines in the codebase, for example, number of goroutines, time taken by each goroutine, etc. The Go method is used to start a new goroutine and track it. The Wait method is used to wait for all goroutines to finish.

Example of usage:

g := NewGoroutineTracker()
g.Go(ctx, "task1", doTask1, arg1, arg2)
g.Go(ctx, "task2", doTask2, arg3)
g.Wait()

Problem: The usage of reflect is unnecessary and can be avoided

Well, that code works, but it uses the reflect package to check the function signature then call the function. It's totally unnecessary, and we can avoid it by changing the usage to:

g := NewGoroutineTracker()
g.Go(ctx, "task1", func() error {
    return doTask1(arg1, arg2)
})
g.Go(ctx, "task2", func() error {
    return doTask2(arg3)
})

The new code will be simpler and has many benefits:

Type safety: No need to check the function signature using reflect. The compiler will do it for us. The original code has potential runtime errors if the function signature does not match the arguments.
Error handling: We can return an error from the function and handle it in the caller. The original code ignores the output of the function.
Readability: The new code is more readable and easier to understand. We can see the function signature and arguments directly in the code.

A better implementation of GoroutineTracker

Here is the refactored code:

func (g *GoroutineTracker) Go(ctx context.Context, fn func() error) {
    g.wg.Add(1)
    id := g.startCaptureTime()
    go func() (err error) {
        defer func() {
            r := recover()
            // capture returned error and panic
            g.endCaptureTime(id, r, err)
            g.wg.Done()
        }()
        // just call the function, no reflect needed
        return fn()
    }()
}

Wait for all goroutines to finish before shutting down

Another use case for GoroutineTracker is for waiting all goroutines to finish before shutting down the application. So we can have 2 types of waiting:

In a function: Waiting for all local goroutines to finish.
When application shutdown: Waiting for all goroutines that started by any GoroutineTracker to finish.

We can implement it by adding a global tracker and making any tracker register its function to the global tracker:

type GlobalTracker struct {
    wg sync.WaitGroup
    // ... some other fields
}
type GoroutineTracker struct {
    parent *GlobalTracker
    wg sync.WaitGroup
    // ... some other fields
}
func (g *GlobalTracker) New() *GoroutineTracker {
    return &GoroutineTracker{parent: g}
}
func (g *GoroutineTracker) Go(ctx context.Context, fn func() error) {
    g.wg.Add(1)            // use both parent and local wg
    g.parent.wg.Add(1)     //   to track the new goroutine
    id := g.startCaptureTime()
    go func() (err error) {
        defer func() {
            // ...
            g.endCaptureTime(id, r, err)
            g.wg.Done()
            g.parent.wg.Done()
        }()

        return fn()
    }()
}
func (g *GlobalTracker) WaitForAll() { g.wg.Wait() }
func (g *GoroutineTracker) Wait()    { g.wg.Wait() }

And we can use WaitForAll() to wait for all goroutines to finish before shutting down the application:

type FooService {
    tracker *GlobalTracker
    // ... some other fields
}
func (s *FooService) DoSomething(ctx context.Context) {
    g := s.tracker.New()
    g.Go(ctx, func() error { return s.doTask1(arg1, arg2) })
    g.Go(ctx, func() error { return s.doTask2(arg3) })
    g.Wait()     // wait for local goroutines, this is optional
}

func main() {
    // some initialization, then start the application
    globalTracker := &GlobalTracker{}
    fooService := FooService{tracker: globalTracker, /*...*/}
    application.Start()

    // wait for all goroutines to finish before shutting down
    <-application.Done()
    globalTracker.Wait()
}

Conclusion

In conclusion, while the original implementation of GoroutineTracker works and can track goroutines, its use of the reflect package to dynamically check and call functions introduces unnecessary complexity and potential runtime errors. By refactoring the code to directly accept function literals, we achieve improved type safety, streamlined error handling, and enhanced readability. This approach leverages Go's compiler-checked type system to ensure compatibility between function signatures and arguments, resulting in more robust and maintainable code. By adopting these changes, we optimize the GoroutineTracker for clarity and reliability, aligning with best practices in Go programming.

Author

I'm Oliver Nguyen. A software maker working mostly in Go and JavaScript. I enjoy learning and seeing a better version of myself each day. Occasionally spin off new open source projects. Share knowledge and thoughts during my journey.

ezpkg.io - Collection of packages to make writing Go code easier

Oliver Nguyen — Thu, 27 Jun 2024 15:33:36 +0000

As I work on various Go projects, I often find myself creating utility functions, extending existing packages, or developing packages to solve specific problems. Moving from one project to another, I usually have to copy or rewrite these solutions. So I created ezpkg.io to have all these utilities and packages in one place. Hopefully, you'll find them useful as well.

The logo is Gopherz - which I created using DALL-E.

Let's look at some problems that these packages are solving.

Handling errors in functions that always return nil errors

For example, let’s look at the following function using strings.Builder from the standard library:

import "fmt"
import "strings"

func SliceToString1[T any](slice []T) string {
    var b strings.Builder
    for _, v := range slice {
        _, err := fmt.Fprint(&b, v)
        if err != nil {
            panic(err)
        }
    }
    return b.String()
}
func SliceToString2[T any](slice []T) (string, error) {
    var b strings.Builder
    for _, v := range slice {
        _, err := fmt.Fprint(&b, v)
        if err != nil {
            return "", err
        }
    }
    return b.String(), nil
}
func SliceToString3[T any](slice []T) string {
    var b strings.Builder
    for _, v := range slice {
        _, _ = fmt.Fprint(&b, v) //nolint:errcheck
    }
    return b.String()
}

In SliceToString1, we add a panic check even though strings.Builder will always return nil error. In SliceToString2, we correctly handle the returned error by making the caller worry about checking an error that never occurs! In SliceToString3, we skip the check because the errors are nil anyway, but we still have to add _, _ = to make the IDE happy and //nolint:errcheck because our company blocks merging any PR that does not pass the golint CI check.

Another way, we could create our utility functions to simplify the code, then copy or write those fprint and must from package to package and project to project:

func SliceToString4[T any](slice []T) string {
    var b strings.Builder
    for _, v := range slice {
        fprint(&b, v)
    }
    return b.String()
}
func fprint(w io.Writer, v any) {
    must(fmt.Fprint(w, v))
}
func must[T any](v T, err error) T {
    if err != nil {
        panic(err)
    }
    return v
}

Import "ezpkg.io/stringz"

Here is how the example is rewritten with ezpkg.io/stringz using stringz.Builder:

import "ezpkg.io/stringz"

func SliceToString[T any](slice []T) string {
    var b stringz.Builder     // change to stringz
    for _, v := range slice {
        b.Print(v)            // look ma, no error!🔥
    }
    return b.String()
}

Other examples include bytez.Buffer and fmtz.State. They share the same interface and include various methods that are ready to use. Let's look at WriteString and its variant WriteStringZ:

package stringz // import "ezpkg.io/stringz"

type Builder strings.Builder

func (b *Builder) unwrap() *strings.Builder {
    return (*strings.Builder)(b)
}
func (b *Builder) WriteString(s string) (int, error) {
    return b.unwrap().WriteString(s)
}
func (b *Builder) WriteStringZ(s string) int {
    n, _ := b.unwrap().WriteString(s)
    return n
}

The WriteString method exposes the original method and keeps the same signature, while the WriteStringZ variant eliminates the need for handling errors. Writing Go code is eazier now!🥰

Sometimes, we just want to skip all the errors to quickly write a simple CLI script

Using typez.CoalesceX, errorz.Must, errorz.Skip, errorz.Validate, and their variants, we can streamline error handling:

import "ezpkg.io/errorz"

func main() {
    var err error
    projectDir := os.Getenv("PROJECT_DIR")
    errorz.ValidateTof(&err, projectDir != "", "no PROJECT_DIR")
    errorz.ValidateTof(&err, len(os.Args) > 1, "must at least 1 arg")

    // panic if any validation fails
    errorz.MustZ(err)

    // get the file path
    jsonFile := os.Args[1] // already check: len(os.Args)>1

    // panic if the file extension is not .json
    if !filepath.IsAbs(jsonFile) {
        jsonFile = filepath.Clean(filepath.Join(projectDir, jsonFile))
    }
    errorz.MustValidate(strings.HasSuffix(jsonFile, ".json"))

    // read the file, skip error if it does not exist
    data, _ := os.ReadFile(jsonFile)

    // default to empty json object
    data = typez.CoalesceX(data, []byte("{}"))

    // process then print the formatted json
    object := errorz.Must(process(data))
    formatted := errorz.Must(json.MarshalIndent(object, "", "\t"))
    fmt.Printf("%s", formatted)
}

Comparing values in tests with diff ignoring spaces

Another day, we are making some changes to a SQL repository method using gorm.io/gorm and gomock. The test code looks like this:

var updateSQL = `UPDATE "company_channels"
SET "updated_at"=$1,"access_token"=$2
WHERE ("company_id" = $3 AND "channel_code" = $4 AND "channel_type" = $5) AND "company_channels"."deleted_at" IS NULL`

dbCtrl.SQLMock.ExpectExec(regexp.QuoteMeta(updateSQL)).
    WithArgs(
        sqlmock.AnyArg(),
        companyChannel.AccessToken,
        companyChannel.CompanyID,
        companyChannel.ChannelCode,
        companyChannel.ChannelType,
    ).WillReturnResult(sqlmock.NewResult(0, 1))

We might encounter this error, which is hard to read and see what is wrong:

Import "ezpkg.io/diffz"

Rewrite the assertion function:

diffz.IgnoreSpace().DiffByChar(actualSQL, expectedSQL)

This provides cleaner output and quickly highlights differences:

Support for tests with random values

Using diffz.Placeholder().AndIgnoreSpaces().DiffByLine() or simply diffz.ByLineZ():

expect := `
color:
  id: ████████-████-████-████-████████████
  name: red
  size: small
  code: #ff0000`
red := `
color:
  id: d56d5f0d-f05d-4d46-9ce2-af6396d25c55
  name: red
  size: small
  code: #ff0000`
green := `
color:
  id: 5b01ec0b-0607-446e-8a25-aaef595902a9
  name: green
  size: small
  code: #00ff00`

fmt.Println("no diff")
fmt.Println(diffz.ByLineZ(red, expect))

fmt.Println("diff")
fmt.Println(diffz.ByLineZ(red, green))

The first diffz.ByLineZ(red, expect) will be considered equal, because of the use of placeholder █. The second diffz.ByLineZ(red, green) will output:

ezpkg.io is in its early stage

These packages are created to enhance the functionality of the standard library and other popular packages. They are intended to be used together with other packages rather than replacing them. The APIs are designed based on my experience working with Go, focusing on simplicity and ease of use. I will try to follow best practices in Go, but not always. I also tend to choose a more performance implementation if possible.

Versioning

All packages are released together with the same version number to simplify management, as they often call each other. When the API evolves, the version number is incremented for all packages.

Why should you NOT use these packages?

More dependencies: These packages will add more dependencies to your project.
Early development: This project is in its early stages and will have API changes. There are other packages that are more mature and offer more features.
Customization: Sometimes, writing your own code allows for better customization. You can also copy code from these packages and modify it to fit your specific needs.

Why should you use these packages?

You find yourself copying the same code over and over.
You are starting a new project and want some simple and easy-to-use packages.
You are learning Go and want to see how some common tasks are implemented.

Stay Tuned

Most packages are usable but the API may change over time. There are a lot of missing utilities that I will add sooner or later. If you need something or want to share your thoughts, feel free to open an issue or start a discussion. I’m curious to see if these packages are making life eazier for you! 👋

Happy coding! 🚀

Author

Latest JavaScript features in 2022

Oliver Nguyen — Sun, 18 Sep 2022 22:29:11 +0000

JavaScript is changing fast with a lot of new and exciting features to help us - developers write code better and better. These features have been released for a while. Let’s check them out!

1. Private fields

Previously, all class fields must be defined in the constructor. And there is no private fields. All fields can be accessed from outside. Of course, there are tricks with scope to make some variables inaccessible. But let’s not talk about them.

class Counter {
  constructor() {
    this.name = 'Counter';
    this.count = 0;
  }

  inc() {
    this.count++;
  }
}

Now we can simply write:

class Counter {
  name = 'Counter';
  #count = 0;  // private field!

  inc() {
    this.#count++;
  }
}

2. Check accessing private fields with in keyword

Private fields come with some problem. When a private field is accessed on an object without it, an error will be thrown. But how to check if an object has it? Use try/catch!

There is now a better way: in keyword.

class Counter {
  name = 'Counter';
  #count = 0;

  static isCounter(obj) {
    return #count in obj;
  }
}

const counter = new Counter();
Counter.isCounter(counter);   // true

3. `at()` method for array and string

We can access an item of an array at position i by writing A[i]. But what if i is negative and we want to get the element from the end of the array? We can write A[A.length + i]. But it’s not very convenient. Another way is use A.slice(i)[0]. But it’s not very efficient.

There is at() method for this:

const A = [2, 4, 6, 8, 10]
A.at(-1)    // 10

const S = "Hello World"
S.at(-1)    // 'd'

Find an item in an array from the end with findLast()

How to find an item in an array? Use find() method. And how to find from the end? Use reverse() and find(). Or write your own function:

const A = [1, 20, 3, 40, 5];

function findBackward(A, predicate) {
  for (let i = A.length-1; i>=0; i--) {
    if (predicate(A[i])) {
      return A[i];
    }
  }
  return -1;
}

findBackward(A, x => x % 10 == 0);    // 40

// be careful with this function!
A.reverse().find(x => x % 10 == 0);   // 40

Now we can use findLast() and findLastIndex() methods:

const A = [1, 20, 3, 40, 5];

A.find(v => v%10 == 0)     // 20
A.findLast(v => v%10 == 0) // 40

A.findIndex(v => v%10 == 0)     // 1
A.findLastIndex(v => v%10 == 0) // 3
A.findLastIndex(v => v == 0)    // -1

Let’s use hasOwn() instead of hasOwnProperty()

There is Object.prototype.hasOwnProperty() method to check if an object has a property as its direct property. But it’s quite cumbersome to use:

let hasOwnProperty = Object.prototype.hasOwnProperty;

if (hasOwnProperty.call(object, 'foo')) {
  console.log('has property foo');
}

But why not just write:

object.hasOwnProperty('foo')

Keep in mind that JavaScript is a dynamic language. We can add a property to any object. So hasOwnProperty() can be shadowed by a property of the object with the same name. To avoid this, we can use hasOwn() method:

if (Object.hasOwn(object, 'foo')) {
  console.log('has property foo');
}

cause is a new property of error

It’s very common to see error handling code like this:

await fetch('https://example.com/data.csv')
  .catch((err) => {
     throw new Error('failed to get: ' + err.message);
  })

What it does is wrapping the original error with a new error. But the original error is lost. Now we can use cause property to store the original error and retrieve it later:

await fetch('https://example.com/data.csv')
  .catch((err) => {
     throw new Error('failed to get', { cause: err })
  })
  .catch((err) => {
     console.log('cause', err.cause)
  })

#! Hashbang is now supported

Hashbang is now supported. And we can run NodeJS script directly in terminal:

#!/usr/bin/env node

'use strict';
console.log(1);

#!/usr/bin/env node

export {};
console.log(1);

I’m Oliver Nguyen. I share about programming, JavaScript, and Go. I created OneSidebar — a browser extension for navigating GitHub. Follow me on Twitter for more content.

Eventually consistency and cache

Oliver Nguyen — Thu, 24 Feb 2022 11:29:28 +0000

What is it?

Eventually consistency is a fancy name of doing something and only expecting the changed state after a while. And there is a guarantee that you will eventually get the expected state. Hence, consistency.

But that won’t work well with caching. And I had to deal with that recently when working with WhatsApp API.

When querying for a list of message templates from WhatsApp, we’ll cache the response for use later. It’s because message templates are not something that changes so frequently. Just cache it to reduce unnecessary API calls.

After creating a template, we won’t see it for a while if we call the listing API. That’s because of, you guessed it, eventually consistency.

Things will get worse because these responded message templates will be cached for a few minutes. Therefore, we won’t see the newly created template soon, even though it may be available on the WhatsApp API earlier.

How to deal with it?

First, let’s simply ignore the cache for, like 1 minute, after creating a new template. That would be enough for the WhatsApp API to return the newly created template.

The next improvement would be reducing the cache expiration time to, like 5s, in the first 1 minute. Let’s call that the catching-up duration. After that, use the original expiration time.

Another improvement would be verifying that the new template is available in the response and starting caching at that point. That would be a bit more complex, as we need to store the template id and more code to extract it from the response.

Finally, while that new template is not available, we can still construct it and manually inject it into our own response.

Recap

In the end, that works well. We can have both cache and eventually consistency playing together, minimizing the waiting time and the number of requests to an external platform! 🎉

Thanks for reading! Would love to hear your thoughts by leaving a comment.

Previously published at olvrng.github.io.

Front-End development without node_modules

Oliver Nguyen — Mon, 08 Feb 2021 14:06:30 +0000

Once upon a time, we could simply put an html and a script file into an FTP server, quickly have a working website and call it a day.

Today, we have to jump through many hoops just to get the right things in the right places. Suppose Alice wants to spend her weekend making a simple to-do app or whatever little idea she enjoys. First, she has to install a big heap of 10k npm packages files. Then she spends a few hours searching how to get this week's js bundler working with the latest typescript with the latest trending UI framework. And a lot frustrating when things don't work or the articles have just outdated. Once she has actually started building the first feature for her little fun app, the weekend has mostly gone!

But things are changing...

1. A little backstory: CommonJS and ES Module

Working with NodeJS, we all get familiar with CommonJS, the standard (read: legacy) way for NodeJS to load dependency code. After installing a module, for example lodash, we can load it into our code by using require('lodash'). This is how NodeJS handles dependency code from the beginning:

const {snakeCase} = require('lodash');

['HelloWorld', 'left pad', 'ECMAScript'].forEach(text => {
  console.log(snakeCase(text));
});

// somewhere in lodash package
function snakeCase(input) {
  // where magic happens
}

exports.snakeCase = snakeCase;

ECMAScript 2015 (ES6) introduced ES Module - an official, standardized module system for JavaScript. It took a while to get here. Nowadays, all major browsers and NodeJS (since v13.2.0) ship support for ES Module by default. ES Module has the advantage of static analysis, tree shaking and asynchronous.

import {snakeCase} from 'lodash';

['HelloWorld', 'left pad', 'ECMAScript'].forEach(text => {
  console.log(snakeCase(text));
});

// somewhere in lodash package
export function snakeCase(input) {
  // where magic happens
}

In NodeJS, to enable ES Module, we have two choices: use .mjs extension or set "type": "module" in package.json. And while most development tools understand ES Module, there are still many incompatibles. For example, TypeScript still does not support outputting to .mjs files. Or Vercel does not work with ES Module. So some transpilers and workarounds are still required. Hopefully the situation will change soon™.

Many packages in NodeJS are already shipped with ES Module files. But many packages are not. At the time of this writing, in the top 10 depended packages on npm, only tslib supports ES Module file by including "exports" in package.json. Many other top packages still don't ship ES Module: lodash, react, moment, request, axios, chalk, commander, express... It's actually not a problem for NodeJS, because NodeJS allows using import to work with both ES Module and CommonJS format.

But browsers don't have that privilege. What if you want to import your favorite node module in the browser? Well, you have to be lucky. At the time of this writing, the recommended way for React to get started in the browser is to include the UMD version in <script> tag, and use the global variable window.ReactDOM:


<script src="https://unpkg.com/react@17/umd/react.development.js"></script>
<script src="https://unpkg.com/react-dom@17/umd/react-dom.development.js"></script>
<script>
  ReactDOM.render(
      <h1>Hello, world!</h1>,
      document.getElementById('root')
  );
</script>

No ES Module for Alice.

2. Skypack

Skypack is a wonderful CDN service that transpiles node packages to be able to work well in the browser. It's backed by Snowpack team. Simply put the package name@version after cdn.skypack.dev and you are ready to go:

<script type="module" src="myscript.js"></script>

// myscript.js
import {snakeCase} from 'https://cdn.skypack.dev/lodash@4';

['HelloWorld', 'left pad', 'ECMAScript'].forEach(text => {
  console.log(snakeCase(text));
});

It just works! Of course, you may ask, there is "lodash-es" that we can import. But many packages don't have their doppelgängers. Or they are not frequently updated. Here, skypack.dev comes to the rescue.

There are still some problems, though. For unclear reasons, some versions did not work. When visiting cdn.skypack.dev/react@16, the React version 17 is served instead. But the future is bright. Alice can now start working right on her app without spending most of her weekend configuring this week's js bundler ...

Side note: I also put my own version at espkg.vercel.app/react@16. You can
use espkg.vercel.app as an alternative until Skypack fixes the problem. Other packages also work, for example, espkg.vercel.app/lodash@4 (give it a little time for building, then the response will be cached by Vercel).

3. Snowpack

Okay, I lied, a bit. TypeScript won't work directly in the browser. You still need more work. Here come the real power of Snowpack: minimal config and remote packages. You don't even have to install node_modules to start working with your little fun app. Simply run 2 setup commands:

yarn global add snowpack
snowpack init

This will give you an empty skeleton snowpack.config.js. Then add the single line config source: 'remote' under packageOptions:

// snowpack.config.js
packageOptions: {
  source: 'remote',
},

That's all! Now run snowpack dev and start adding your index.html and myscript.ts (yes, it's TypeScript):

<!doctype html>
<html lang="en">
<head>
  <title>My little app</title>
</head>
<body>
<script type="module" src="myscript.js"></script>
</body>
</html>

// myscript.ts
import {snakeCase} from 'lodash';

const words: string[] = ['HelloWorld', 'left pad', 'ECMAScript'];
words.forEach(text => {
  console.log(snakeCase(text));
});

It just works! 🎉 Look ma, no node_modules! No package.json! We even got TypeScript and hot-reload for free. Yay!

Recap

The example code can be downloaded here: gist.github.com/olvrng. There are other configs on snowpack.config.js that you may need. Let's save that for another day. Now start tinkering with your app and spend your precious time on the most valuable feature! 🚀🚀

P.S.

Oh, but Alice wants to use less. No worry, just a single line config... I swear! She could add a file mystyle.less and one more line for snowpack.config.js. Everything will be okay. Well, this time she must remember to run yarn add snowpack-plugin-less! Only this time...

// snowpack.config.js
plugins: [
  'snowpack-plugin-less',
],

// myscript.ts
import {snakeCase} from 'lodash';
import './style.less'; // the less file

const words: string[] = ['HelloWorld', 'left pad', 'ECMAScript'];
words.forEach(text => {
  console.log(snakeCase(text));
});

Thank you for reading! And don't forget my little page espkg.vercel.app.

µjson — A minimal JSON parser and transformer in Go

Oliver Nguyen — Fri, 08 Jan 2021 09:19:13 +0000

µjson is a minimal JSON parser and transformer that works on unstructured (and trusted) JSON. It works by parsing input and calling the given callback function when encountering each item.

Motivation

Sometimes we just want to make some minimal changes to a JSON document or do some generic transformations without fully unmarshalling it. For example, removing blacklisted keys from response JSON. Why spend all the cost on unmarshalling into a map[string]interface{} just to immediately marshal it again. The following code is taken from StackOverflow:

{
  "responseHeader": {
    "status": 0,
    "QTime": 0,
    "params": {
      "q": "solo",
      "wt": "json"
    }
  },
  "response": {
    "numFound": 2,
    "start": 0,
    "docs": [
      { "name": "foo" },
      { "name": "bar" }
    ]
  }
}

With µjson, we can quickly write a simple transformation to remove "responseHeader" completely from all responses, once and forever. More on that later.

The original scenario that leads me to write the package is because of int64. When working in Go and PostgreSQL, I use int64 (instead of string) for ids because it’s more effective and has enormous space for randomly generated ids. It’s not as big as UUID, 128 bits, but still big enough for production use. In PostgreSQL, those ids can be stored as bigint and being effectively indexed. But for JavaScript, it can only process integer up to 53 bits (JavaScript has BigInt but that’s a different story, and using it will make things even more complicated).

So we need to wrap those int64s into strings before sending them to JavaScript. In Go and PostgreSQL, the JSON is {"order_id": 12345678} but JavaScript will see it as {"order_id": "12345678"} (note that the value is quoted). In Go, we can define a custom type and implement the json.Marshaler interface. But in PostgreSQL, that’s just not possible or too complicated. I wrote a service that receives JSON from PostgreSQL and converts it to be consumable by JavaScript. The service also removes some blacklisted keys or does some other transformations (for example, change orderId to order_id).

So I wrote a simple JSON parser and transformer. It can:

Print all keys and values in order
Reformat input
Remove all whitespaces
Remove blacklisted keys
Wrap int64s into strings before handing to JavaScript
Extract some values
… and more.

Important: Behavior is undefined for invalid JSON, use on trusted input only! For untrusted input, you might want to run it through json.Valid() before handing it to µjson.

Let’s see how µjson work by examples:

1. Print all keys and values in order

The callback function is called when an object key/value or an array key is encountered. It receives 3 params in order: level, key and value.

level is the indentation level of the JSON, if you format it properly. It starts from 0. It increases after entering an object or array and decreases after leaving.
key is the raw key of the current object or empty otherwise. It can be a double-quoted string or empty.
value is the raw value of the current item or a bracket. It can be a string, number, boolean, null, or one of the following brackets: { } [ ]. Values will never be empty.

It’s important to note that key and value are provided as raw. Strings are always double-quoted. It’s there for keeping the library fast and ignoring unnecessary operations. For example, when you only want to reformat the output JSON properly; you don’t want to unquote those strings and then immediately quote them again; you just need to output them unmodified. And there are ujson.Unquote() and ujson.AppendQuote() when you need to get the original strings.

When processing arrays and objects, first the open bracket ([, {) will be provided as value, followed by its children, and the close bracket (], }). When encountering open brackets, you can make the callback function return false to skip the object/array entirely.

package main    

import "fmt"    
import "github.com/olvrng/ujson"    

func main() {   
    input := []byte(`{  
        "id": 12345,    
        "name": "foo",  
        "numbers": ["one", "two"],  
        "tags": {"color": "red", "priority": "high"},   
        "active": true  
    }`) 
    ujson.Walk(input, func(level int, key, value []byte) bool { 
        fmt.Printf("%2v% 12s : %s\n", level, key, value)    
        return true 
    })  
}

 0             : {
 1        "id" : 12345
 1      "name" : "foo"
 1   "numbers" : [
 2             : "one"
 2             : "two"
 1             : ]
 1      "tags" : {
 2     "color" : "red"
 2  "priority" : "high"
 1             : }
 1    "active" : true
 0             : }

0. The simplest examples

To easily get an idea on level, key and value, here are the simplest examples:

package main

import "fmt"
import "github.com/olvrng/ujson"

func main() {
    input0 := []byte(`true`)
    ujson.Walk(input0, func(level int, key, value []byte) bool {
        fmt.Printf("level=%v key=%s value=%s\n", level, key, value)
        return true
    })
    // output:
    //   level=0 key= value=true

    input1 := []byte(`{ "key": 42 }`)
    ujson.Walk(input1, func(level int, key, value []byte) bool {
        fmt.Printf("level=%v key=%s value=%s\n", level, key, value)
        return true
    })
    // output:
    //   level=0 key= value={
    //   level=1 key="key" value=42
    //   level=0 key= value=}

    input2 := []byte(`[ true ]`)
    ujson.Walk(input2, func(level int, key, value []byte) bool {
        fmt.Printf("level=%v key=%s value=%s\n", level, key, value)
        return true
    })
    // output:
    //   level=0 key= value=[
    //   level=1 key= value=true
    //   level=0 key= value=]
}

In the first example, there is only a single boolean value. The callback function is called once with level=0, key is empty and value=true.

In the second example, the callback function is called 3 times. Two times for open and close brackets with level=0, key is empty and value is the bracketed character. The other time for the only key with level=1, key is "key" and value=42. Note that the key is quoted and you need to call ujson.Unquote() to retrieve the unquoted string.

The last example is like the second, but with an array instead. Keys are always empty inside arrays.

2. Reformat input

In this example, the input JSON is formatted with correct indentation. As processing the input key by key, the callback function reconstructs the JSON. It outputs each key/value pair in its own line, prefixed with spaces equal to the param level. There is a catch, though. Valid JSON requires commas between values in objects and arrays. So there is ujson.ShouldAddComma() for checking whether a comma should be inserted.

package main

import "fmt"
import "github.com/olvrng/ujson"

func main() {
    input := []byte(`{"id":12345,"name":"foo","numbers":["one","two"],"tags":{"color":"red","priority":"high"},"active":true}`)

    b := make([]byte, 0, 1024)
    err := ujson.Walk(input, func(level int, key, value []byte) bool {
        if len(b) != 0 && ujson.ShouldAddComma(value, b[len(b)-1]) {
            b = append(b, ',')
        }
        b = append(b, '\n')
        for i := 0; i < level; i++ {
            b = append(b, '\t')
        }
        if len(key) > 0 {
            b = append(b, key...)
            b = append(b, `: `...)
        }
        b = append(b, value...)
        return true
    })
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s\n", b)
}

{
    "id": 12345,
    "name": "foo",
    "numbers": [
        "one",
        "two"
    ],
    "tags": {
        "color": "red",
        "priority": "high"
    },
    "active": true
}

There is a built-in method ujson.Reconstruct() when you want to remove all the whitespaces.

3. Remove blacklisted keys

This example demonstrates removing some keys from the input JSON. The key param is compared with a pre-defined list. If there is a match, the blacklisted key and its value are dropped. The callback function returns false for skipping the entire value (which may be an object or array). Note that the list is quoted, i.e. "numbers" and "active" instead of number and active. For more advanced checking, you may want to run ujson.Unquote() on the key.

package main

import "bytes"
import "fmt"
import "github.com/olvrng/ujson"

func main() {
    input := []byte(`{
        "id": 12345,
        "name": "foo",
        "numbers": ["one", "two"],
        "tags": {"color": "red", "priority": "high"},
        "active": true
    }`)

    blacklistFields := [][]byte{
        []byte(`"numbers"`), // note the quotes
        []byte(`"active"`),
    }
    b := make([]byte, 0, 1024)
    err := ujson.Walk(input, func(_ int, key, value []byte) bool {
        for _, blacklist := range blacklistFields {
            if bytes.Equal(key, blacklist) {
                // remove the key and value from the output
                return false
            }
        }

        // write to output
        if len(b) != 0 && ujson.ShouldAddComma(value, b[len(b)-1]) {
            b = append(b, ',')
        }
        if len(key) > 0 {
            b = append(b, key...)
            b = append(b, ':')
        }
        b = append(b, value...)
        return true
    })
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s\n", b)
}

{"id":12345,"name":"foo","tags":{"color":"red","priority":"high"}}

As you see in the output, "numbers" and "active" are removed.

4. Wrap int64 in string

This is the original motivation behind µjson. The following example finds keys ending with _id" ("order_id", "item_id", etc.) and converts their values from numbers to strings, by simply wrapping them in double-quotes.

For valid JSON, values are never empty. We can test the first byte of value (value[0]) to get its type:

n : Null
f , t : Boolean
0...9 : Number
" : String, see ujson.Unquote()
[ , ] : Array
{ , } : Object

In this case, we check value[0] within 0…9 to see whether it’s a number, then insert double-quotes.

package main

import "bytes"
import "fmt"
import "github.com/olvrng/ujson"

func main() {
    input := []byte(`{"order_id": 12345678901234, "number": 12, "item_id": 12345678905678, "counting": [1,"2",3]}`)

    suffix := []byte(`_id"`) // note the ending quote "
    b := make([]byte, 0, 256)
    err := ujson.Walk(input, func(_ int, key, value []byte) bool {
        // Test for keys with suffix _id" and value is an int64 number. For valid json,
        // values will never be empty, so we can safely test only the first byte.
        shouldWrap := bytes.HasSuffix(key, suffix) && value[0] > '0' && value[0] <= '9'

        // transform the input, wrap values in double quotes
        if len(b) != 0 && ujson.ShouldAddComma(value, b[len(b)-1]) {
            b = append(b, ',')
        }
        if len(key) > 0 {
            b = append(b, key...)
            b = append(b, ':')
        }
        if shouldWrap {
            b = append(b, '"')
        }
        b = append(b, value...)
        if shouldWrap {
            b = append(b, '"')
        }
        return true
    })
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s\n", b)
}

{"order_id":"12345678901234","number":12,"item_id":"12345678905678","counting":[1,"2",3]}

After processing, the numbers in "order_id" and "item_id" are quoted as strings. And JavaScript should be happy now! 🎉 🎉

Recap

You can start using it by import "github.com/olvrng/ujson". The source code is short and easy to read. Feedback is welcome 👋

DEV Community: Oliver Nguyen

iter.json: A Powerful and Efficient Way to Iterate and Manipulate JSON in Go

1. Iterating JSON

2. Building JSON

3. Formatting JSON

4. Adding line numbers

5. Adding comments

6. Filtering JSON and extracting values

7. Filtering JSON and returning a new JSON

8. Editing values

Conclusion

Author

Refactor: GoroutineTracker with unnecessary usage of reflect

Problem: The usage of reflect is unnecessary and can be avoided

A better implementation of GoroutineTracker

Wait for all goroutines to finish before shutting down

Conclusion

Author

ezpkg.io - Collection of packages to make writing Go code easier

Handling errors in functions that always return nil errors

Import "ezpkg.io/stringz"

Sometimes, we just want to skip all the errors to quickly write a simple CLI script

Comparing values in tests with diff ignoring spaces

Import "ezpkg.io/diffz"

Support for tests with random values

ezpkg.io is in its early stage

Versioning

Why should you NOT use these packages?

Why should you use these packages?

Stay Tuned

Author

Latest JavaScript features in 2022

1. Private fields

2. Check accessing private fields with in keyword

3. at() method for array and string

Eventually consistency and cache

What is it?

How to deal with it?

Recap

Front-End development without node_modules

1. A little backstory: CommonJS and ES Module

2. Skypack

3. Snowpack

Recap

P.S.

µjson — A minimal JSON parser and transformer in Go

Motivation

Let’s see how µjson work by examples:

1. Print all keys and values in order

0. The simplest examples

2. Reformat input

3. Remove blacklisted keys

4. Wrap int64 in string

Recap

3. `at()` method for array and string