Fast Museum Searches: Go Concurrency and Caching

Matt DeMartino — Wed, 11 Feb 2026 16:13:16 +0000

Intro

Hello!
In this post I plan on talking about how I receive artwork data and store it. Speed is paramount when creating successful websites, and making multiple API calls can really slow down a website. Since Museum Passport searches multiple Museum APIs, optimization is key to create a positive user experience. I will explain my strategies for making Museum Passport feel as fast as possible, and how I hope to make it even faster in the future.

Go Concurrency

One of the main reasons I chose Go for Museum Passport was its great concurrency features. One of the newer concurrency features of Go is the errgroup. Errgroup is basically a high level way of running multiple goroutines at once, rather than making channels to handle errors can get hard to manage. Instead, errgroup cleanly handles possible errors for you. Although errgroup returns only the first error it receives, it ends up being far cleaner than standard channels.

artworks := make([]*models.SingleArtwork, len(currentSearch))
g := new(errgroup.Group)
g.SetLimit(10)
for i, id := range searchIDs {
g.Go(func() error {
artwork, err := artworkAPICall(id)
if err != nil {
return err
}
artworks[i] = artwork
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}

NOTE: Go 1.22 fixed loop variable capture issues

The very first line of code involves preallocating an empty slice to store the returned data. This is a safe way of allowing multiple goroutines to write to the same slice. There is no fear of resizing issues when each goroutine is preassigned its own spot in the slice.

Another great feature of errgroup is the ability to easily set the maximum number of goroutines occurring at once. Without SetLimit, Go could theoretically start running a hundred API calls at once, which would probably not end well! SetLimit(10) ensures at most 10 concurrent calls are made, preventing you from overwhelming the API.

Caching Artwork

Making concurrent API calls is great, but would it be efficient to make the same calls repeatedly? That is where caching comes in. I decided on first focusing on using local memory for artwork data, as it is relatively low in size. The main downside of storing in local memory though is that when the application is restarted, all that saved data is lost.

artwork, exists := m.Cache.GetArtwork(fmt.Sprintf("met-%d", id))
if exists {
return &artwork, nil
}

Before deciding to make a call to a museum's API, the function first checks to see if the artwork already exists in the cache, therefore eliminating an API call. Say you're searching for artwork by Van Gogh, and 3 of the 10 returned IDs are already stored in the cache, only 7 API calls are necessary to return the needed data, speeding up the search experience.

When working with a relatively small cache (50 artworks at roughly 16KB total is negligible), local memory works well. However, I plan on implementing Redis or a database in the future to handle cache persistence across restarts and create a more production-scale application.

Conclusion

By combining Go's errgroup for concurrent API calls with in-memory caching, Museum Passport can serve data to users quickly without overwhelming external APIs. These optimizations create a responsive search experience while using APIs responsibly.
Thank you for reading!

Normalizing API Data from Multiple Sources

Matt DeMartino — Wed, 14 Jan 2026 17:16:21 +0000

Hello!
This blog documents my journey in creating Museum-Passport, an educational website exploring the world of art museums. One of the main goals of the site is to aggregate works of art from multiple museums and display them in one place. Many major museums offer public APIs for interacting with their collections. Using Go, my goal is to use these APIs to populate Museum Passport with information on art and cultures from around the world and throughout history. I chose Go for this project because of its excellent support for concurrent API calls (goroutines), clean data modeling (structs), and flexible interfaces.

My first goal when starting out was to properly strategize how to use these different APIs in an efficient and logical manner. This involved researching the various APIs and noting what information they can provide.

The problem: APIs aren't standardized

When browsing the various museum APIs available, the first thing that stood out is how differently they are structured. The Met's API has all the expected information: name, artist, location, date, etc. But the API of the Harvard Museums includes information such as colors found in the art, descriptions, and even what exhibits they have been featured in.

Not only do they differ in the levels of information provided, but fields that serve the same purpose in each API can have differing names. Examples include dated vs. objectDate, constituents vs people, and so on. The question was: how do I take these differently structured APIs and normalize their responses?

Creating an artwork struct

My solution was creating a standardized artwork struct to normalize museum API data.

type SingleArtwork struct {
ID           string
ArtworkTitle string
ArtistName   string
DateCreated  string
ArtMedium    string
ImageLarge   string
ImageSmall   string
Museum       string
}

Normally an ID would be an integer, but it is quite likely that an ID number might be used in multiple museums. I decided to include a short-form version of the originating museum's name. SingleArtwork IDs therefore might look like "met-23443" or "harvard-3245". This identifies which museum the artwork comes from, while also ensuring their ID is unique. All that's left is to take the fields from the API and insert them into a new SingleArtwork struct. Thankfully this is quite simple, and only involves using the API field as the value for the SingleArtwork field.

func (m *MetClient) NormalizeArtwork(receivedArt MetSingleArtwork) models.SingleArtwork {
return models.SingleArtwork{
ID: fmt.Sprintf("met-%d", receivedArt.ObjectID),
ArtworkTitle: receivedArt.Title,
ArtistName: receivedArt.ArtistDisplayName,
DateCreated: receivedArt.ObjectDate,
ArtMedium: receivedArt.Medium,
ImageLarge: receivedArt.PrimaryImage,
ImageSmall: receivedArt.PrimaryImageSmall,
Museum: m.GetMuseumName(),
}
}

Here's what a normalized response looks like:

{
  "ID": "met-436105",
  "ArtworkTitle": "The Death of Socrates",
  "ArtistName": "Jacques Louis David",
  "DateCreated": "1787",
  "ArtMedium": "Oil on canvas",
  "ImageLarge": "https://images.metmuseum.org/CRDImages/ep/original/DP-13139-001.jpg",
  "ImageSmall": "https://images.metmuseum.org/CRDImages/ep/web-large/DP-13139-001.jpg",
  "Museum": "The Metropolitan Museum of Art"
}

Takeaway

What makes me most proud is the fact that I thought about how to tackle this beforehand. Like many developers, my first instinct is to dive in headfirst and begin coding without any real direction. I decided to approach every aspect of this project from a slower, more intentional angle. I brainstormed potential roadblocks and then researched how to overcome them.

I hope you continue with me on this journey as I continue to blog about my experiences!
Matt

DEV Community: Matt DeMartino