Clever Caching | Stock Analysis app using Netlify's fine-grained cache control

This is a submission for the Netlify Dynamic Site Challenge: Clever Caching

What I Built

I have build a Stock Analysis tool which shows price history, financial data and related News articles for any NYSE listed stock.
Github Link

Cache Mechanisms used in the Project

• Netlify-Vary headers
• Netlify-CDN-Cache-Control headers
• SWR (Stale while revalidate)

On-demand Cache Invalidation mechanism using

• purgeCache() helper function
• Netlify-Cache-Tag headers to purge cache by tags

Demo

A 10 min video explaining the app and code base.

Link to app: https://market-lens-app.netlify.app/

Cache Status Screen shots

1. Initial load of a stock page will have Cache-Status as "Netlify Edge"; fwd=miss
2. Further load of the same page will have Cache-Status as "Netlify Edge"; hit

Platform Primitives

Tell us how you leveraged Netlify Cache Control.

To begin with I firstly configured cache as manual in the netlify.toml file for the paths I want to cache.

[[edge_functions]]
  function='news'
  path= '/api/news/:symbol'
  cache='manual'

[[edge_functions]]
  function='financial'
  path= '/api/financial'
  cache='manual'

In the above file you can see something called symbol, this stands for stock ticker symbol (Example AAPL is the symbol for Apple INC ). This symbol plays a key role for the caching mechanism in this project. Let see how.

For the sake of learning I have used two approaches for querying data
1: Ticker symbol in Path Params.

api/news/AAPL -> will fetch all news article related to Apple stock

2: Ticker symbol in Query Params.

api/financial?symbol=AAPL -> will fetch financial data for Apple stock

1. Caching Response when dynamic value in Path params

For fetching the news data the ticker symbol is passed in the URL's path param. The response header looks like below.

const headers = {
  'Content-Type': 'application/json',
  'Cache-Control': 'public, max-age=0, must-revalidate',
  'Netlify-CDN-Cache-Control':'public, max-age=120, stale-while-revalidate=300',
  'Cache-Tag': `${symbol}, ${symbol}-news`,
};

I have used Netlify-CDN-Cache-Control with stale-while-revalidate
The stale-while-revalidate in the Netlify cache header allows cached content to be served even after it expires (max-age), while simultaneously refreshing the cache in the background. In this case, it allows serving stale content for up to 300 seconds while a new version is fetched and revalidated.

When the dynamic variable (the stock symbol) is used as a path param you dont have to do much as the symbol is part of the url path.

/api/news/AAPL -> Will have Apple news article Cached
/api/news/MSFT -> -> Will have Microsoft news Article Cached

The challenge is when symbol is passed in query params. lets see how its cached.

2. Caching with Netlify-Vary | Dynamic value in Query params

When the symbol is passed in query params then you have to add a custom header Netlify-Vary in-order to cache the response. The response header looks like below.

export default async (req: Request, context: Context) => {
  const parsedUrl = new URL(req.url);
  const queryParams = Object.fromEntries(parsedUrl.searchParams.entries());
  const symbol = queryParams.symbol
...
const headers = {
      'Content-Type': 'application/json',
      'Access-Control-Allow-Origin': '*',
      'Cache-Control': 'public, max-age=0, must-revalidate',
      'Netlify-CDN-Cache-Control':
        'public, max-age=604800 , stale-while-revalidate=2592000',
      'Netlify-Cache-Tag': `${symbol}, ${symbol}-financial`,
      'Netlify-Vary': 'query=symbol',
    };

This header object is similar to the one we saw above except that we have added an additional header field Netlify-Vary with value query=symbol. This makes sure to cache response taking the symbol query param into consideration.

/api/financial?symbol=MSFT -> Will cache Microsoft financial data
/api/financial?symbol=TSLA-> Will cache Tesla's financial data

More on Netlify-Vary by query params

On-demand Cache Purging

I have used Netlify's purgeCache() helper to implement on-demand cache invalidation. I have made this function accessible under the path /manual-purge
Cache invalidation is one of the hardest things to do in Software development but I'm pretty impressed how Netlify has made it easy to implement with the helper function and cache tags.

Cache Tags

You may have noticed a field in my response headers labeled Netlify-Cache-Tag. This field is crucial for managing cache invalidation behavior effectively.

// news.ts
const header ={
....
'Netlify-Cache-Tag': `${symbol}, ${symbol}-news`,
...
}
// financial.ts
const header ={
....
'Netlify-Cache-Tag': `${symbol}, ${symbol}-financial`,
...
}

The use of ticker symbol as tag names allows us to target specific company data for purging without affecting other company data.

The below tables might help you understand how responses are tagged

URL	Cache tags
/api/news/MSFT	MSFT, MSFT-news
/api/financial?symbol=MSFT	MSFT, MSFT-financial
/api/news/TSLA	TSLA, TSLA-news

This granular approach ensures that updates or modifications related to particular stocks or financial information are accurately purged without affecting other data.

Example

1. /manual-purge?tag=MSFT Will purge all cached response related to Microsoft i.e News, Financial and Price data
2. /manual-purge?tag=MSFT-news This will purge only the news data related to Microsoft while other cached data related to Microsoft or any other company is not disturbed. What's surprising is that the code to purge by tags is relatively simple.

  const url = new URL(req.url);
  const cacheTag = url.searchParams.get('tag');
  await purgeCache({
    tags: [cacheTag],
  });

Below image shows how cached response related to Apple stock are purged

Below image shows how only cached response related to Microsoft news are purged

Personal Hurdles

I have been searching API providers for stock market data and
settled with Polygon io as the free plan allows 5 calls per minute.

To be honest, this limitation guided my cache strategy to prioritize minimizing external API calls. By serving cached data for previously accessed stocks, the app successfully avoids redundant calls to the same company.

Conslusion

If you have reached this part of the article it would be great if you to leave a like ❤️ for the post as it would motivate me to experiment more stuffs like this.
As someone implementing edge functions and caching for the first time I found Netlify's documentation comprehensive, sparing me the need to rely heavily on YouTube tutorials. The cache-status in response headers simplified debugging not to mention the real time logs in Netlify console. The ability to group and purge cache using Netlify-cache-tags struck me as a particularly cool feature. Looking forward to use them in real life application.

Thanks to Dev.to team and Netlify for the Hackathon. I learned something new.