You might not be using json.Decoder correctly in golang

TL;DR: prevailing "secondary source" wisdom (ie: blog posts) about json.Decoder don't demonstrate the proper way to use it.

---

This post is a follow up to my (kinda lengthy) deep dive into what I thought was a bug in golang's json.Decoder pkg.

Instead, I realized that generally speaking, json.Decoder can be misunderstood - which may lead to unintended consequences. In this post, I will demonstrate a safer pattern that ought to be used instead of the prevailing wisdom.

Googling: "json.decoder example golang"

I ran a few google searches queries using some permutation of the following:

json.decoder example golang

The results were your standard mix of documentation from golang.org, blog posts on medium.com/random mom&pop devsites (like this one!), and a few stackoverflow threads.

Fortunately, results from golang.org are highly ranked - while it may be a bit harder to parse through golang's src code, the documentation is fairly thorough and more importantly: correct. (Personally, I would have preferred some additional context in the docs expounding on some of the gotchas I discuss on my other post but I digress)

Some of the threads I observed in Stack Overflow that referenced json.Decoder pulled directly from the docs (and therefore were also correct). Other's (probably) pulled from medium/other blog post sites similar to the ones I found googling around and were inaccurate/advocating incorrect usage.

For example, on Medium, et al - I saw a wide array of posts (such as this, this, this or this) that suggested using json.Decoder in some way, shape, or form similarly to this:

func main() {
    jsonData := `{
"email":"[e-mail]",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile Profile
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    // ...
}

(Example pulled from Tutorial: How to work with JSON data in Go)

The Problem

On the surfact, this code looks sound. I forklifted the src into go playground and ran it.

package main

import (
    "encoding/json"
    "strings"
)

func main() {
    jsonData := `{
"email":"[e-mail]",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile map[string]interface{}
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    // ...
}

(Note, changed profile to map[string]interface{} to make the code run)

...It works! Great. But, what happens if we fubar the JSON string?

package main

import (
    "encoding/json"
    "strings"
)

func main() {
    jsonData := `{
"email":"[e-mail]",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}THIS IS INTENTIONALLY MALFORMED NOW`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile map[string]interface{}
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    // ...
}

(playground)

...It works!

Wait...WTF?!.

This code should not work at all! Our JSON string is clearly malformed and we expect - based on the logic - the code to panic.

This is the entire issue in a nutshell. I expound in detail in my other post but in one (kinda long) sentence:

json.Decoder.Decode was implemented for parsing streaming JSON data, meaning it will always traverse the JSON string until it finds a satisfactory, closing bracket (I use the term satisfactory here because it does use a stack to keep track of inner brackets).

So, in order to detect the malformed json, we must actually run this logic in a loop - like so:

package main

import (
    "encoding/json"
    "io"
    "strings"
)

func main() {
    jsonData := `{
"email":"[e-mail]",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}THIS IS INTENTIONALLY MALFORMED NOW`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile map[string]interface{}
    for {
        err := decoder.Decode(&profile)
        if err != nil {
            panic(err)
        }
        if err == io.EOF {
            break
        }
    }

    // ...
}

(playground)

Note the key diff here:

var profile map[string]interface{}
for {
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    if err == io.EOF {
        break
    }
}
// ...

The Fix

From the golang docs, this example says it best:

// This example uses a Decoder to decode a stream of distinct JSON values.
func ExampleDecoder() {
    const jsonStream = `
    {"Name": "Ed", "Text": "Knock knock."}
    {"Name": "Sam", "Text": "Who's there?"}
    {"Name": "Ed", "Text": "Go fmt."}
    {"Name": "Sam", "Text": "Go fmt who?"}
    {"Name": "Ed", "Text": "Go fmt yourself!"}
`
    type Message struct {
        Name, Text string
    }
    dec := json.NewDecoder(strings.NewReader(jsonStream))
    for {
        var m Message
        if err := dec.Decode(&m); err == io.EOF {
            break
        } else if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%s: %s\n", m.Name, m.Text)
    }
    // Output:
    // Ed: Knock knock.
    // Sam: Who's there?
    // Ed: Go fmt.
    // Sam: Go fmt who?
    // Ed: Go fmt yourself!
}

(sauce)

In this usage, we create a new json.Decoder instance from the NewDecoder method and then continually loop and try to decode a chunk of our JSON string until we detect the end of the string (sucessfully breaking out of the loop) or an error.

I would take this a step further and only prefer to use json.Decoder when I am specifically working with streaming JSON data.

At any rate, this the The Way. Please keep this in mind going forward should you choose to use json.Decoder for your JSON string parsing needs.

A few notes

• A big caveat I'd like to point out is that I did not perform any rigorous analysis of golang examples in the wild when I was inspired to write this post. My analysis comes largely from anecdotal examples I observed while researching my other deep dive article about the nature of json.scanner and json.Decoder.Decode.
• I certianly am not being critical of the medium/mom&pop blogs I linked to earlier, instead I am merely pointing out that it is very easy to assume that the trivial usage of json.Decoder.Decode presented on those posts is generally correct when in actuality this is not the case.
• I think someone (maybe me? perhaps you?) ought to open a PR against godocs to clarify gotchas/differences between json.Decoder usage vs json.Unmarshal

---

PS: this article is x-posted from my dev blog

Comments

[Dennis]

Good write-up. Shouldn't these two be reversed?
if err != nil {
panic(err)
}
if err == io.EOF {
break
}
Also, if you are reading data from memory, the preferred method for doing so is json.Unmarshal, not Decode.

[Hugo Soto]

What about using json.Valid instead of major refactor?? I don't know if it is too expensive to use but i think its worth the try.