In Go, the encoding/json package offers built-in JSON marshaling and unmarshaling capabilities, allowing us to effortlessly convert Go data structures to JSON and vice versa. However, there are scenarios where the default behavior falls short, such as handling complex data structures, managing custom field names, or dealing with special data types. This article aims to provide an introduction on how JSON can be handled in Go, and as always, with a bit of customization.
Disclaimer: I tend to make lots of mistakes on a daily basis. If you’ve noticed one, I’d be more than grateful if you let me know.
- All the code snippets can be found in my Github repo.
Understanding JSON Marshaling and Unmarshaling
JSON (JavaScript Object Notation) is a lightweight data interchange format widely used for communication between web services and applications. It represents data as key-value pairs, arrays, and nested structures, providing a simple and human-readable format for data exchange.
In Go, the encoding/json package is a powerful tool for working with JSON data. It offers the Marshal function to convert Go data structures into JSON, and the Unmarshal function to deserialize JSON into Go data structures. These functions handle most common cases automatically, mapping Go struct fields to JSON object properties and the other way around. However, when dealing with more complex scenarios or requiring fine-grained control over the serialization and deserialization process, JSON tags as well as custom marshaling and unmarshaling functions shine.
JSON Tags
The way that Go handles JSON revolves around structs and field tags. Structs represent the data structure to be serialized or deserialized, with each field of the struct corresponding to a property in the JSON representation. Field tags, defined using the json struct tag, provide metadata that controls the serialization and deserialization behavior. By leveraging field tags, we can specify custom names for JSON properties, handle null values, control field omission, and more. Let’s see an example of these tags:
type Person struct {
Name string `json:"full_name"`
Age int `json:"-"`
Email string `json:"email,omitempty"`
}
Name is mapped to the JSON property full_name
Age will be fully ignored (won’t show up no matter what)
Email will be mapped to email and the omitempty is a reserved flag which ensures that if the email field is empty (e.g. contains the zero value of its type) it’s not going to show up after being serialized into JSON
Let’s see how these tags work in action:
pp := []Person{
{
Name: "first person",
Age: 20,
Email: "first@person.com",
},
{
Name: "second person",
Age: 25,
Email: "",
},
}
b, _ := json.MarshalIndent(pp, "", "\t")
fmt.Println(string(b))
[
{
"full_name": "first person",
"email": "first@person.com"
},
{
"full_name": "second person"
}
]
- Note that omitempty if used solely as a JSON tag will be considered the field name rather than being treated as a flag.
type Person struct {
Name string `json:"omitempty"`
}
pp := []Person{
{
Name: "first person",
},
{
Name: "second person",
},
}
b, _ := json.MarshalIndent(pp, "", "\t")
fmt.Println(string(b))
[
{
"omitempty": "first person",
},
{
"omitempty": "second person"
}
]
NOTE: **Duplicate names will be failed to get serialized silently (e.g. no errors, yet **none of the duplicate fields appear in the output), although you’ll probably be warned by a static code analysis tool or a linter.
JSON tags (or let’s say any custom tag) can be obtained at runtime via reflection like below:
func getFieldTags(t reflect.Type) map[string][]string {
if t.Kind() != reflect.Struct {
panic(fmt.Sprintf("t should be struct but is %s", t))
}
tags := make(map[string][]string)
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
jTags := strings.SplitN(f.Tag.Get("json"), ",", -1)
tags[f.Name] = jTags
}
return tags
}
func main() {
tags := getFieldTags(reflect.TypeOf(Person{}))
fmt.Printf("%+v\n", tags) // map[Age:[-] Email:[email omitempty] Name:[full_name]]
}
Custom JSON Marshaling/Unmarshaling
In order to implement a custom JSON marshal/unmarshaler you have to implement the json.Marshaler or json.Unmarshaler interfaces accordingly. Let’s have a look at these interfaces:
// Marshaler is the interface implemented by types that
// can marshal themselves into valid JSON.
type Marshaler interface {
MarshalJSON() ([]byte, error)
}
// Unmarshaler is the interface implemented by types
// that can unmarshal a JSON description of themselves.
// The input can be assumed to be a valid encoding of
// a JSON value. UnmarshalJSON must copy the JSON data
// if it wishes to retain the data after returning.
//
// By convention, to approximate the behavior of Unmarshal itself,
// Unmarshalers implement UnmarshalJSON([]byte("null")) as a no-op.
type Unmarshaler interface {
UnmarshalJSON([]byte) error
}
Let’s say that we have to make a call to some remote API in order to obtain information about certain products, and for the sake of the example let’s assume that the API response is as follows:
{
"products": [
"prod A",
"prod B",
"prod C"
],
"prices": [
99.99,
199.99,
299.99
]
}
We know (and expect) that the same indexes contain information about a certain product, e.g. prices[0] belongs to products[0]. in order to better handle this data we want to deserialize (unmarshal) the API response into the following structures:
type Product struct {
Name string
Price float64
}
type Cargo struct {
Products []Product
}
In order for that to work, we need to declare an UnmarshalJSON method for our destination (here Cargo) struct:
func main() {
// our imaginary API
d := GetProductPrices()
cargo := &Cargo{}
err := json.Unmarshal(d, cargo)
if err != nil {
log.Fatal("failed to unmarshal to cargo:", err)
}
fmt.Printf("%+v\n", cargo)
}
func (c *Cargo) UnmarshalJSON(d []byte) error {
...
}
In order to have fine grained control over how our data is deserialized we can temporarily unmarshal it into any desired intermediary structure:
func (c *Cargo) UnmarshalJSON(d []byte) error {
// desired intermediary structure
type temp struct {
Products []string `json:"products"`
Prices []float64 `json:"prices"`
}
tmp := &temp{}
err := json.Unmarshal(d, tmp)
if err != nil {
return fmt.Errorf("failed to marshal JSON in temp: %w", err)
}
...
}
Or we can even use fully extensible data structure like map[string]any :
func (c *Cargo) UnmarshalJSON(d []byte) error {
// desired intermediary structure
tmp := make(map[string]any)
err := json.Unmarshal(d, &tmp)
if err != nil {
return fmt.Errorf("failed to marshal JSON in temp: %w", err)
}
...
}
Having that the deserialized data is ready for further processing, the next step is to transform it into whatever structure we want:
func (c *Cargo) UnmarshalJSON(d []byte) error {
type temp struct {
Products []string `json:"products"`
Prices []float64 `json:"prices"`
}
tmp := &temp{}
err := json.Unmarshal(d, tmp)
if err != nil {
return fmt.Errorf("failed to marshal JSON in temp: %w", err)
}
if len(tmp.Prices) != len(tmp.Products) {
return fmt.Errorf("length of products (%d) and prices (%d) does not match", len(tmp.Products), len(tmp.Prices))
}
for i := 0; i < len(tmp.Products); i++ {
c.Products = append(c.Products, Product{Name: tmp.Products[i], Price: tmp.Prices[i]})
}
return nil
}
Finally we can expect to have a nicely organized data like below:
{
"Products": [
{
"Name": "prod A",
"Price": 100
},
{
"Name": "prod B",
"Price": 200
},
{
"Name": "prod C",
"Price": 300
}
]
}
While I was experimenting with these stuff I came up with a somewhat interesting question, something that might be bothering you right now as well:
In the case that our structs do not implement json.Marshaler or json.Unmarshaler interfaces, Go still finds a way to serialize from and deserialize to our structures. How is that possible?
Well, under the hood, what Go really does is that it looks to see if the given struct implements either of those interfaces. “How?” you might ask. Well, again, all thanks to reflection:
// json/decode.go
// check if a certain type implements "Unmarshaler" (v is reflect.Value)
if v.Type().NumMethod() > 0 && v.CanInterface() {
if u, ok := v.Interface().(Unmarshaler); ok {
return u, nil, reflect.Value{}
}
// json/encode.go
// check if a certain type implements "Marshaler" (t is reflect.Type)
marshalerType = reflect.TypeOf((*Marshaler)(nil)).Elem()
if t.Kind() != reflect.Pointer && allowAddr && reflect.PointerTo(t).Implements(marshalerType) {
return newCondAddrEncoder(addrMarshalerEncoder, newTypeEncoder(t, false))
}
In case the type fails to pass the test (e.g. does not implement neither json.Marshaler nor json.Unmarshaler ) Go falls back on something like a plan B.
To be honest I mostly use custom mashaling/unmarshaling capabilities to permenantly change JSON files embedded in the program, by first reading the JSON file, deserializing it into whatever structure I want and then serialize it back to a new .json file. Something like this:
type CustomStruct struct {
// whatever field I want at the end
}
func (c *CustomStruct) UnmarshalJSON(d []byte) error {
// custom JSON unmarshalling
}
func main() {
// read, change, write
d, err := os.ReadFile("old.json")
if err != nil {
log.Fatal("failed to read file:", err)
}
st := &CustomStruct{}
err = json.Unmarshal(d, st)
if err != nil {
log.Fatal("failed to unmarshal:", err)
}
b, err := json.MarshalIndent(st, "", "\t")
if err != nil {
log.Fatal("failed to marshal struct:", err)
}
err = os.WriteFile("new.json", b, 0644)
if err != nil {
log.Fatal("failed to write new file:", err)
}
}
Note that there are two major things that I’ve not considered here, one is reading the file in a streaming manner (so I don’t flood my memory by reading the file as a whole at once) and the other is partial unmarshaling in JSON. We might take a look at these subjects later.
Conclusion
Custom JSON marshaling and unmarshaling in Go allows us to overcome the limitations of the default JSON handling capabilities provided by the standard library. By leveraging custom marshalers and unmarshalers, we can finely tune the serialization and deserialization processes, making our code more expressive, efficient, and tailored to specific use cases. We have explored various aspects of custom JSON handling in Go, from implementation techniques to real-world examples. I hope you’ve enjoyed our journey. Thanks for your support in advance!
Top comments (0)