Build a web API client in Go, Part 2: Deserializing the data

In part 1 of this tutorial, we got an API key for the ClimaCell API, used it to connect to ClimaCell in cURL to get back some JSON weather data, and then set up the same thing in Go. Where we left off in commit 1, we made an http.Request to a URL on the ClimaCell API's hourly forecast endpoint, got back some data in an http.Response, and read it into a byte slice.

The next thing we need to do in order to build the Go app the Cambridge Fresh Pond sloths are building, is to take that HTTP response data and convert it to a Go struct that our code can work with, so we need to deserialize it.

📚 Understanding the API data we're deserializing

The idea of deserialization, is that you're taking data encoded in some format like JSON, XML, CSV, or Gob, and translating it to an object that our code is able to work with.

Luckily, Go gives us a flexible encoding/json package for converting JSON data to Go structs. But before we define our struct for weather data, we gotta make the first stop of any web API client design: visiting the API documentation! Very often, the people maintaining a website's API will give you samples of the API request and response data formats, information on how to authenticate with the API, and explanations of the fundamentals to working with the API.

You can find ClimaCell's API reference here. Let's scroll through and look for information on how ClimaCell represents their weather data.

Starting at the Data Layers > Weather section, there is a list of all the fields of data about the weather that the API can return. For example there's temp for what temperature it is, and wind_speed for how fast the wind is blowing. It is worth noting that for any given weather sample, if data for a field is unavailable, that field's value on the weather sample will be null.

Below that in the left bar is links to all the different URLs we can work with on ClimaCell's API, or as they're called in API terminology, API endpoints. Check out the /weather/forecast/hourly endpoint's documentation, which is the endpoint we want.

On the right, you can see a sample of a 200 (OK) response's JSON. Write down a few notes about the data format, and then compare what you noticed to my list below.

Some points of interest I spotted about this data format are:

• The response data is formatted as an array of JSON objects; one object for each weather sample in the forecast.
• The fields' names are in snake_case.
• For each kind of field you can request, the data is inside a JavaScript object saying the field's value in a weather sample, and what units of measure that value is in.

Knowing these details will help us design the Go struct we'll be converting our JSON data to. On the observation in the last bullet point, although for this tutorial we only really care about deserializing the temp field, we know that every numeric piece of data except the lat and lon coordinates is inside a "units, and nullable numeric value" object. So if we make a struct to deserialize those objects, like this:

type FloatValue struct {
    Value *float64
    Units string
}

Then we could use that struct as the type to fields on our weather sample struct not just for the temperature, but for any numeric fields in a weather sample!

📦 Making a struct to deserialize our data to

We have a good understanding of the format of the data we're working with, so let's make a Weather struct we can deserialize our data. If you remember from our last tutorial, our forecast response, with only temperature requested, was an array of objects like this one:

  {
     "lon":              -71.146,
     "lat":              42.3826,
     "temp":             { "value": 5, "units": "C" },
     "observation_time": { "value": "2020-04-24T15:00:00.000Z" },
  }

lat and lon are float64's, and as we talked about in the last section, temp would be that FloatValue struct we made. And observation_time is a timestamp inside a JSON object, so we'd want a similar TimeValue struct. Since every weather sample has to have a timestamp, we'll assume that TimeValue is not nullable.

So in addition to an overall Weather struct, let's make our FloatValue and TimeValue structs. Make a folder under the tea-temperature folder titled climacell, and add this Go code to weather.go:

package climacell

import (
    "time"
)

type FloatValue struct {
    Value *float64
    Units string
}

type NonNullableTimeValue struct { Value time.Time }

Sweet, now with FloatValue and TimeValue defined in the climacell package, and the float64 type defined by Go itself, we have types representing every field we requested. So let's go ahead and put them together on a Weather struct in climacell/weather.go:

type Weather struct {
    Lat             float64
    Lon             float64
    Temp            *FloatValue
    ObservationTime NonNullableTimeValue
}

One thing to note is that we made Temp a *FloatValue. As we saw with our cURL request, only get back a weather field from the ClimaCell API if you specifically ask for it, so if a user didn't request the temperature in a forecast, for example, temp should not be on the JSON response, and therefore should be nil on the Weather structs for that response.

Let's try out our struct. In main.go we already had some bytes we can work with from that HTTP response, so let's try deserializing them! At the top of the file, import encoding/json and "github.com/{your github name}/tea-temperature/climacell", and then in the part right after where we run ioutil.ReadAll, add this code to deserialize and work with our data:

    var weatherSamples []climacell.Weather
    if err := json.Unmarshal(responseBytes, &weatherSamples); err != nil {
        log.Fatalf("error deserializing weather data")
    }

    for _, w := range weatherSamples {
       if w.Temp != nil && w.Temp.Value != nil {
         log.Printf("The temperature at %s is %f degrees %s\n",
           w.ObservationTime.Value, *w.Temp.Value, w.Temp.Units)
       } else {
         log.Printf("No temperature data available at %s\n",
           w.ObservationTime.Value)
       }
    }

Run go install and then tea-temperature, and the output will look like:

2020/04/19 23:22:47 The temperature at 0001-01-01 00:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:22:47 The temperature at 0001-01-01 00:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:22:47 The temperature at 0001-01-01 00:00:00 +0000 UTC is 5.000000 degrees C

Strange, we have the temperature data, but the time field is the zero value for time.Time. In the actual cURL response, we did see a timestamp, so it's definitely there. We need just a small tweak on our Weather struct to get observation_time deserialized correctly. Our current progress is in commit 2.

🏷 Adding struct tags for JSON deserialization

The reason why the JSON response's observation_time didn't get added to our Weather struct was because observation_time, and all other data fields on the response, are in snake_case. This presents a problem because Go's JSON deserialization by default checks for camelCase JSON object fields.

That means if our weather data instead was {"observationTime": {"value": "2020-04-19T13:00:00.000Z"}}, Go's JSON deserialization would detect the camelCase observationTime object field and correctly move its data over to our Weather struct's ObservationTime field.

Because ClimaCell uses a snake_case data format, though, we do need to tell Go's deserialization to look for an observation_time field. We can do that by adding struct tags, which are annotations you can add to Go structs to indicate things like which JSON field corresponds to the struct field, or how to represent a given struct field in a database.

Using a JSON struct tag looks like this:

type Weather struct {
    Lat             float64
    Lon             float64
    Temp            *FloatValue
    ObservationTime TimeValue `json:"observation_time"`
}

The json:"observation_time" is our struct tag, and it tells the encoding/json package, "if you see a field titled observation_time when you're deserializing a JSON object to a Weather struct, put its data in the struct's ObservationTime field".

Though when I define an API client, even if we don't need struct tags on every field, I like adding them to all fields we deserialize from JSON so it's really explicit how a Go struct maps to a JSON object. So let's change the structs in weather.go to:

type FloatValue struct {
    Value *float64 `json:"value"`
    Units string   `json:"units"`
}

type NonNullableTimeValue struct {
    Value time.Time `json:"value"`
}

type Weather struct {
    Lat             float64              `json:"lat"`
    Lon             float64              `json:"lon"`
    Temp            *FloatValue          `json:"temp"`
    ObservationTime NonNullableTimeValue `json:"observation_time"`
}

Now run go install again, and when you run tea-temperature, your output should look like this:

2020/04/19 23:35:12 The temperature at 2020-04-24 13:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:35:12 The temperature at 2020-04-24 14:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:35:12 The temperature at 2020-04-24 15:00:00 +0000 UTC is 5.000000 degrees C

Great work! We deserialized our response data! Now, there's one quick refactor we can do to really streamline JSON deserialization, and that is use a json.Decoder! Our progress so far is in commit 3.

💍 Using a json.Decoder

Previously, we've been converting our JSON responses to Weather structs by first using ioutil.ReadAll to convert the response body to a byte slice. Then we converted that byte slice into our slice of weather structs with json.Unmarshal.

It would be really convenient if we could do all the deserialization in one step, converting straight from the io.ReadCloser response body, to our slice of Weather structs, with no middleman.

That's where the json.Decoder type comes in! Instead of doing this:

    responseBytes, err := ioutil.ReadAll(res.Body)
    if err != nil {
        log.Fatalf("error reading HTTP response body: %v", err)
    }

    var weatherSamples []climacell.Weather
    if err := json.Unmarshal(weatherBytes, &weatherSamples); err != nil {
        log.Fatalf("error deserializing weather data")
    }

You can pass any io.Reader, such as our response body, into json.NewDecoder to build a json.Decoder. Then, its Decode method handles all the deserialization. So converting our JSON to Weather structs would now look like this:

    var weatherSamples []climacell.Weather
    d := json.NewDecoder(res.Body)
    if err := d.Decode(&weatherSamples); err != nil {
        log.Fatalf("error deserializing weather data")
    }

Not only is that shorter, but it's also more memory-efficient, so json.Decoder.Decode is a common idiom for deserializing HTTP responses in Go.

Our progress so far is in commit 4.

We looked through the ClimaCell API to learn about the hourly forecast endpoint and its response format, and then we made a Go type to represent ClimaCell weather data. As you can see, most of the work building this struct turned out to be studying API documentation. This is the case for any API you're building a client for. Building a client is all about doing research (both reading docs and sending requests to experiment with endpoints) so that your client's code can correctly handle all the API's details.

Up to this point, we've been sending requests with a Go net/http client, so in the next tutorial, we're gonna take our requests and data deserialization, and put it all together in our own climacell.Client type. That type will build on top of http.Client to make a Go type that's tailored for a streamlined experience for Go developers working with the ClimaCell API.

Until next time, 🦥 STAY SLOTHFUL! 🌺

Comments

[Donald Wilson]

Hi,

Excellent tutorial!

Taking your mantra, Stay Slothful, to heart, I thought you might find following website helpful:

mholt.github.io/json-to-go/

DW