DEV Community: Tomás Murúa

Introducing SerpApi's New Markdown Output

Tomás Murúa — Mon, 24 Aug 2026 19:59:59 +0000

We’re proud to announce the launch of Markdown Output, a new feature designed to deliver real-time search data in a format optimized for LLMs and AI agents. It is available on every search, on every API, and it comes back from the same structured search that JSON does. This is SerpApi’s effort to adapt and improve our product for LLMs and AI agents. Based on our benchmarks, the new output gives you roughly half the tokens of JSON on average and up to 90% for some APIs. If you already use our APIs, you can implement it right away; no configuration is needed, and it works automatically across our 100+ APIs.

SerpApi: Markdown Output for AI Agents

In this post, we’ll show what the new Markdown Output looks like on a real search and explain when to use it and when to stay on JSON.

Getting Markdown results does not require a new endpoint. Any existing search can return it in three ways:

# Query parameter
curl "https://serpapi.com/search?engine=google&q=coffee&api_key=YOUR_KEY&output=md"

# Route extension
curl "https://serpapi.com/search.md?engine=google&q=coffee&api_key=YOUR_KEY"

# Accept header
curl -H "Accept: text/markdown" "https://serpapi.com/search?engine=google&q=coffee&api_key=YOUR_KEY"

Why Markdown for LLMs

Markdown is a plain-text format with headers, tables, links, and lists that you can read raw or render as structure. In the last few years, it also became the default language of LLMs. Models are trained on enormous amounts of it (documentation, READMEs, forums); chat interfaces render their answers from it, and OpenAI's own prompt engineering guide recommends Markdown headers and lists to communicate hierarchy to the models.

We took that signal from our users and from the industry, everyone converging on Markdown as the common input and output format for LLMs and agents, and adapted our product to it. Every design choice in the new output serves one goal, to fit more useful data in your model's context window. Let's see each one with a real search for coffee on Google.

SerpApi Playground - SerpApi

Token Efficient

Take the first organic result from a coffee search on Google. This is everything JSON carries for it:

{
  "position": 1,
  "title": "Coffee",
  "link": "https://en.wikipedia.org/wiki/Coffee",
  "redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://en.wikipedia.org/wiki/Coffee&ved=2ahUKEwiOsKzK96uWAxXmQjABHecdCtsQFnoECCcQAQ",
  "displayed_link": "https://en.wikipedia.org › wiki › Coffee",
  "thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQs1E6FVWWD12JBVWlsho0fGc7GxAR50Pf030IPyxJNl5A9xYPmgt_1&usqp=CAE&s",
  "favicon": "https://serpapi.com/images/i/iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b-u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz_6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1-flswMu1EQZ3xlE7lK8eKDkjtwE-crBMV-wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk_9ryhFMAAAAASUVORK5CYII.png",
  "snippet": "Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content;",
  "snippet_highlighted_words": [
    "a beverage brewed from roasted ground coffee beans"
  ],
  "about_this_result": {
    "source": {
      "description": "Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content;",
      "source_info_link": "https://en.wikipedia.org/wiki/Coffee",
      "icon": "https://serpapi.com/images/i/iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b-u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz_6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1-flswMu1EQZ3xlE7lK8eKDkjtwE-crBMV-wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk_9ryhFMAAAAASUVORK5CYII.png"
    },
    "languages": [
      "en"
    ],
    "regions": [
      "US"
    ]
  },
  "about_page_link": "https://www.google.com/search?q=About+https://en.wikipedia.org/wiki/Coffee&tbm=ilp",
  "about_page_serpapi_link": "https://serpapi.com/search.json?engine=google_about_this_result&google_domain=google.com&q=About+https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FCoffee",
  "source": "Wikipedia"
}

And the same result as a Markdown table row:

## Organic Results

| Position | Title | Thumbnail | Snippet | About Page Link | Source | Read More Link | Date |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | [Coffee](https://en.wikipedia.org/wiki/Coffee) | ![Thumbnail](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQs1E6FVWWD12JBVWlsho0fGc7GxAR50Pf030IPyxJNl5A9xYPmgt_1&usqp=CAE&s) | Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content; | https://www.google.com/search?q=About+https://en.wikipedia.org/wiki/Coffee&tbm=ilp | Wikipedia |  |  |

Nothing the model needs is missing. The position, title, thumbnail, snippet, and source are all kept in the output, and the row costs a fraction of the tokens the object did. Across the full coffee response, the JSON weighs 24,723 tokens and the markdown 6,435, a 74% cut.

Here are some examples:

Token savings across SerpApi APIs

Native Markdown Links

In JSON a result splits its data across separate fields:

"title": "Coffee",
"link": "https://en.wikipedia.org/wiki/Coffee",
"displayed_link": "https://en.wikipedia.org › wiki › Coffee"

In Markdown that is one cell:

[Coffee](https://en.wikipedia.org/wiki/Coffee)

The model gets the title and the URL already bound together.

No Noise

The same JSON result also carries a redirect_link (a 150-character Google tracking URL), a 320-character favicon URL, and internal metadata objects:

"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://en.wikipedia.org/wiki/Coffee&ved=2ahUKEwiOsKzK96uWAxXmQjABHecdCtsQFnoECCcQAQ",
"favicon": "https://serpapi.com/images/i/iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b-u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz_6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1-flswMu1EQZ3xlE7lK8eKDkjtwE-crBMV-wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk_9ryhFMAAAAASUVORK5CYII.png"

None of that helps a model answer a question about coffee. The Markdown Output strips them automatically, and you can check the table row above; none of these fields made it in. The title, link, snippet, and source remain.

Structured Output

Markdown does not flatten the response into prose. Each result array becomes a table, and the nine organic results of our coffee search arrive as nine rows under one header:

## Organic Results

| Position | Title | Thumbnail | Snippet | About Page Link | Source | Read More Link | Date |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | [Coffee](https://en.wikipedia.org/wiki/Coffee) | ![Thumbnail](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQs1E6FVWWD12JBVWlsho0fGc7GxAR50Pf030IPyxJNl5A9xYPmgt_1&usqp=CAE&s) | Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content; | https://www.google.com/search?q=About+https://en.wikipedia.org/wiki/Coffee&tbm=ilp | Wikipedia |  |  |
| 2 | [Fresh Roasted, Single Origin Coffee Beans](https://creaturecoffee.co/collections/coffee?srsltid=AfmBOoo5rAxqBgQmreq6MaHX-U_rw-P5UI-90YLvKI1oRGTosk_eXwMU) |  | Creature Coffee roasts single origin coffee beans from around the globe right here in Austin, Texas. | https://www.google.com/search?q=About+https://creaturecoffee.co/collections/coffee?srsltid=AfmBOoo5rAxqBgQmreq6MaHX-U_rw-P5UI-90YLvKI1oRGTosk_eXwMU&tbm=ilp | Creature Coffee Co |  |  |
| 3 | [Coffee Shops Austin, TX - Last Updated August 2026](https://www.yelp.com/search?find_desc=Coffee+Shops&find_loc=Austin%2C+TX) |  | Coffee Shops Austin, TX · 1. Mazúr Coffee · 2. Palomino Coffee · 3. Idlewild Coffee · 4. Desnudo Coffee · 5. Flora Coffee · 6. Daydreamer Coffee. 3.9 ... | https://www.google.com/search?q=About+https://www.yelp.com/search?find_desc=Coffee+Shops&find_loc=Austin%2C+TX&tbm=ilp | Yelp | https://www.yelp.com/search?find_desc=Coffee+Shops&find_loc=Austin,+TX#:~:text=Coffee%20Shops%20Austin%2C%20TX&text=1.%20Maz%C3%BAr%20Coffee&text=2.%20Palomino%20Coffee&text=3.%20Idlewild%20Coffee&text=4.%20Desnudo%20Coffee&text=5.%20Flora%20Coffee&text=6.%20Daydreamer%20Coffee,3.9%20(25%20reviews) |  |
| 4 | [r/Coffee](https://www.reddit.com/r/Coffee/) |  | This is a place to share and talk about your favorite coffee roasters or beans. recommendations, reviews, etc. | https://www.google.com/search?q=About+https://www.reddit.com/r/Coffee/&tbm=ilp | Reddit · r/Coffee |  |  |
| 5 | [Cuvée Coffee: Always Something Brewing. Austin, TX](https://cuveecoffee.com/?srsltid=AfmBOorjX3_f5tAt5zldI8Vqr4pNmmfh2FRWlZcVVMEGXhowAfzLZ_Jt) |  | At Cuvée, coffee isn't a job, it's a passion. Since 1998, we've become known for roasting exceptional, ethically sourced, sustainable coffee. | https://www.google.com/search?q=About+https://cuveecoffee.com/?srsltid=AfmBOorjX3_f5tAt5zldI8Vqr4pNmmfh2FRWlZcVVMEGXhowAfzLZ_Jt&tbm=ilp | Cuvée Coffee |  |  |
| 6 | [Coffee - The Nutrition Source](https://nutritionsource.hsph.harvard.edu/food-features/coffee/) |  | Various polyphenols in coffee have been shown to prevent cancer cell growth in animal studies. Coffee has also been associated with decreased estrogen levels, a ... | https://www.google.com/search?q=About+https://nutritionsource.hsph.harvard.edu/food-features/coffee/&tbm=ilp | The Nutrition Source | https://nutritionsource.hsph.harvard.edu/food-features/coffee/#:~:text=Type%202%20Diabetes |  |
| 7 | [Texas Coffee Traders](https://texascoffeetraders.com/) |  | No information is available for this page. | https://www.google.com/search?q=About+https://texascoffeetraders.com/&tbm=ilp | Texas Coffee Traders |  |  |
| 8 | [33 Best Coffee Shops In Austin, Sorted By Neighborhood](https://www.atasteofkoko.com/visit-austin/coffee-shops-in-austin) |  | Best Coffee Shops in Central Austin · Hank's · Civil Goat Coffee Roasters (Central) · Merit Coffee (Central) · Bennu Coffee · Terrible Love. | https://www.google.com/search?q=About+https://www.atasteofkoko.com/visit-austin/coffee-shops-in-austin&tbm=ilp | A Taste of Koko | https://www.atasteofkoko.com/visit-austin/coffee-shops-in-austin#:~:text=Best%20Coffee%20Shops%20in%20Central,Austin&text=Hank's&text=Civil%20Goat%20Coffee%20Roasters%20(Central)&text=Merit%20Coffee%20(Central)&text=Bennu%20Coffee&text=Terrible%20Love | Apr 22, 2026 |
| 9 | [Try the new gold standard of coffee at home!](https://icoffee.co/) |  | At Industry Coffee, we take great care in every step of our coffee making process. From selecting the best beans, to roasting, brewing, and blending our ... | https://www.google.com/search?q=About+https://icoffee.co/&tbm=ilp | Industry Coffee | https://icoffee.co/#:~:text=At%20Industry%20Coffee%2C%20we%20take,committed%20to%20quality%20and%20consistency. |  |

The same happens for shopping results, hotel listings, and any other result array.

The response also opens with a YAML frontmatter block carrying the search parameters and metadata:

---
search_metadata:
  id: 6a853ba34c437ff2e1aa1265
  status: Success
  json_endpoint: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.json"
  markdown_endpoint: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.md"
  pixel_position_endpoint: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.json_with_pixel_position"
  created_at: "2026-08-19T05:14:11.488Z"
  processed_at: "2026-08-19T05:14:11.497Z"
  google_url: "https://www.google.com/search?q=Coffee&oq=Coffee&uule=w+CAIQICIaQXVzdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM&hl=en&gl=us&sourceid=chrome&ie=UTF-8"
  raw_html_file: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.html"
  total_time_taken: 1.91
search_parameters:
  engine: google
  q: Coffee
  location_requested: Austin, Texas, United States
  location_used: Austin,Texas,United States
  google_domain: google.com
  hl: en
  gl: us
  device: desktop
---

So even in text form you still have the search id, status, and the endpoints to fetch the same search again in either format.

When to Choose Markdown or JSON?

Both formats come from the same search, so this is not a migration. It is a per-call choice.

Markdown fits when:

You feed results straight into an LLM prompt or a RAG context
An agent consumes search results as a tool response
You want a quick human-readable digest of a search
Tokens are the budget you optimize for

JSON fits when:

Your code extracts fields programmatically (prices as numbers, coordinates, ratings)
You handle pagination, filters, or metadata downstream
You store results in a database with a fixed schema
You need every field, including the ones Markdown strips

Simply put: if a model reads it, use Markdown. If code reads it, use JSON.

Using the JSON Restrictor

Before Markdown Output, the tool for cutting response size was the JSON Restrictor, and it is still the right tool when you stay on JSON. You pass a field selector with the request, and SerpApi filters the response server-side before sending it back. We have used it for exactly this purpose in earlier posts, like connecting a local LLM to web search data, where trimming the JSON was the difference between fitting the context window or not.

SerpApi: JSON Restrictor

Cutting Tokens While Staying on JSON

If your pipeline needs structured fields but not all of them, the restrictor alone gets you most of the way. We'll request only the two fields an agent might rank on, position and title:

curl "https://serpapi.com/search?engine=google&q=Coffee&location=Austin,+Texas,+United+States&api_key=YOUR_KEY&json_restrictor=organic_results[].position,organic_results[].title"

The response is the whole search reduced to this:

{
  "organic_results": [
    {
      "position": 1,
      "title": "Coffee"
    },
    {
      "position": 2,
      "title": "Cuvée Coffee: Always Something Brewing. Austin, TX"
    },
    {
      "position": 3,
      "title": "Coffee Shops Austin, TX - Last Updated August 2026"
    },
    {
      "position": 4,
      "title": "r/Coffee"
    },
    {
      "position": 5,
      "title": "Coffee - The Nutrition Source"
    },
    {
      "position": 6,
      "title": "Texas Coffee Traders"
    },
    {
      "position": 7,
      "title": "50 Best Austin Coffee Shops You Should Try"
    },
    {
      "position": 8,
      "title": "Buy Coffee, Tea, Powders Online | The Coffee Bean & Tea ..."
    },
    {
      "position": 9,
      "title": "History of coffee - NCA"
    }
  ]
}

You keep exact types and programmatic access, and drop every section your code never touches. The difference with Markdown Output is what each one removes. The restrictor cuts entire fields and sections you name, while Markdown keeps every section but re-renders it in a lighter format. One subtracts data, the other changes its shape.

Stacking Both

You don't have to pick one. Pass json_restrictor on a Markdown request, and SerpApi first filters the sections, then renders only what is left as Markdown:

curl "https://serpapi.com/search.md?engine=google&q=Coffee&location=Austin,+Texas,+United+States&api_key=YOUR_KEY&json_restrictor=organic_results"

The response is just the Organic Results table from earlier, nothing else. On our coffee search, the four levels look like this:

Request	Tokens
Full JSON	24,723
JSON + `json_restrictor=organic_results`	8,486
Full markdown	6,435
Markdown + `json_restrictor=organic_results`	1,298

That is a 95% reduction from the original JSON, and you chose exactly which part of the search enters the model's context. For an agent that only reasons over organic results, everything else was noise anyway.

Conclusion

Markdown Output is live today on every SerpApi plan, at no extra cost, across all 100+ APIs. Same searches, same credits, and around 50% fewer tokens on average and up to 90% for some APIs.

The fastest way to see what it does for you is to take an API you already use, add output=md to one request, and count the tokens before and after.

Other Resources

Measuring World Cup Attention: What the data says about Vozinha

Tomás Murúa — Sun, 09 Aug 2026 02:13:34 +0000

During the 2026 World Cup, Cape Verde played for the first time, and its goalkeeper Vozinha had the kind of attention that nobody could have imagined before the event started.

His Instagram following stood at roughly 27,000 before the tournament. Then Cape Verde held Spain to a goalless draw, with Vozinha stopping all eight shots on target from the 23 Spain produced. In the days after that match his following passed 14 million. By the end of the tournament it had grown past 29 million.

Vozinha's story raised a question. How many smaller versions of that story were buried across 48 teams and 104 matches?

So during the tournament, we tracked search interest, news coverage, and Instagram followers for a watchlist of players to answer a simple question. Beyond the trophy, who actually captured the world's attention?

This blog is about what the data showed, the player stories hiding in it, and what any of it can and cannot tell you.

The final attention leaderboard in Sports Pulse

Everything in this post comes from Sports Pulse, a live dashboard I built on SerpApi's APIs to track this, the same data teams use for sponsorship scouting, media monitoring, and market research. Every number below has a page there where you can see it for yourself.

Sports Pulse

The Question That Turned Into a Dashboard

The World Cup writes down everything that happens on the pitch. Every goal, every save, every minute played. What it does not write down is who the world went looking for while it happened. That record is scattered across search queries, news archives, and follower counts, three things measured in different units and stored in different shapes. The match report tells you Vozinha played four games. It does not tell you the world went looking for him afterwards.

Vozinha’s World Cup stats

So I built a pipeline that assembles that second record. For every player it produces a single attention score from 0 to 100, built from three signals. Search interest, news coverage, and follower growth, each compared against the rest of the field. The score answers one narrow question.

Relative to everyone else at this tournament, how much attention did this player draw during the event?

It is not a measure of how good a player is, and it is not a measure of fame. A player who was already massively popular before the tournament and stayed that way will score low, because the score is built around movement. And it never says why something happened. A spike after a match is a spike after a match.

I scored a watchlist of 39 hand-picked players rather than the 1,248 players spread across the 48 squads. Scoring all of them means measuring hundreds of players nobody searched for, which buries the signal in noise. So I picked the players most likely to draw attention, based on their profiles going in, minutes played, and how far their team went. Less data, better data.

The 39-player watchlist

Reading the Board

The board does not rank the best players at the World Cup. It ranks who moved the most while it was happening, and the players at the top got there in completely different ways.

The Top Six

The top six by attention score

Lamine Yamal

Yamal finishes first with a score of 73.6, and he is the only player on the board who is strong in all three signals at once. The highest search interest of the tournament, 727 articles, and 13.7 million new followers across Spain's eight-match run to the title.

Yamal's page in Sports Pulse

That is the sanity check. A measure of tournament attention that did not pull Yamal near the top would be measuring something else or completely wrong. What makes him unusual is not the total; it is that he has no weak column.

Vozinha

Vozinha finishes fourth with an attention score of 56.4, across four matches.

Vozinha's page Sports Pulse

He gained 29.5 million followers during the window, the largest movement in the field by a distance nobody else is close to. His whole score rests in one column, and that is the pattern a scoreboard cannot show you and a raw follower count cannot contextualize, because a follower count on its own has no field to compare against.

The per-match Instagram follower graph makes it concrete. The Spain match added 14.2 million followers; the Argentina match in the final added 11.8 million new followers to the Vozinha account.

Vozinha follower growth by match

Lionel Messi

Messi is the most useful player on this watchlist for understanding what the score actually measures.

He has the most news coverage of anyone at the tournament with 783 articles covering him, more than Yamal. His search interest is the second highest on the watchlist. And his follower growth scores exactly zero.

Messi page Sports Pulse

The zero is not missing data, and it is not because he stood still. Messi gained 8.9 million followers during the tournament, more than most players on the watchlist. But on a base of 515 million that is a 0.5% move, and in a field where Vozinha's following multiplied more than a thousand times over, a 0.5% move lands below the middle of the pack and scores nothing.

Messi's follower growth by match

That is the metric behaving exactly as designed and it still looks strange written down. Messi finishing sixth is not a claim that five players had a bigger tournament than him. It is a claim that five players moved more than he did, which is a different sentence, and the difference is the whole point of the exercise.

Haaland and Olise

Vozinha's pattern is not a one-off. Haaland finishes second after gaining 32.8 million followers across six matches.

Haaland page Sports Pulse

And Olise finishes fifth with 6 million gained from a far smaller base. Both maxed out the same follower signal Vozinha did.

Olise page Sports Pulse

Three of the top five got there mostly on follower growth, and that is a finding about tournaments, not about the metric. While the competition is running, the audience moves faster than the press does. Reporters filed a few hundred articles about Vozinha over four weeks. Tens of millions of people tapped follow. His story shows up in follower counts first and everywhere else later.

What the Attention Turned Into

The attention score stops at the tournament window. Vozinha's story did not.

Within weeks of the final he signed with Colo Colo, one of the biggest clubs in Chile, and reports linked him with offers from several other teams, including rumors of a move to Inter Miami to share a locker room with Messi. Sponsorship offers came after. And searches for Cape Verde itself climbed alongside its goalkeeper, a country of half a million people that most of the new audience could not have placed on a map in May.

Cape Verde search interest

All of that happened to a 40-year-old goalkeeper, at an age when most careers are already over. The data in this post cannot tell you what caused any of it. What it can show is the moment the audience arrived, tens of millions of people in two match windows, and everything above followed while the world was watching.

What This Cannot Tell You

Three honest limits, so the numbers above are read as what they are.

It does not explain why. The score measures attention during the tournament. A player whose following grew after a strong performance grew after a strong performance. What produced that growth is outside anything these APIs can see.
It only knows 39 players. Every score is relative to that hand-picked field and would change if the field changed.
It measures attention, not merit. A player can have an outstanding tournament in near silence. The score would show a low number, and it would be right about attention and say nothing about football.

How We Measured This

Everything above comes from one pipeline built on four SerpApi engines. The Google Sports API supplies the skeleton, the tournament's dates, the squads, and all 104 matches, so nobody maintains a match list by hand. On top of that skeleton, three signals are measured for every player on the watchlist over a window taken from the tournament's own dates. The Google Trends API gives search interest, the Google News API gives coverage volume, and the Instagram Profile API gives follower growth rather than raw totals, which is the difference between measuring who is already famous and measuring who is becoming famous.

The three signals arrive in three incompatible units, so each is converted to the same scale before combining into a single 0 to 100 attention score. A player with a gap in one signal is scored on the signals that exist rather than punished with a zero.

How the attention score is measured

How the conversion works and how each engine is queried in practice is a story of its own, and it did not fit here.

Pointing It at the Next Tournament

Nothing in the pipeline knows it is looking at football. The window comes from the event's dates, the player list from the event's structure, and the score from comparing whoever is on the watchlist. Swap the event and the watchlist, and the same three signals work for the Champions League, the NBA, the NFL, and other sports events.

Conclusion

Yamal topped the board by being strong everywhere. Messi put up two of the biggest signals at the tournament next to a zero. And a 40-year-old goalkeeper from a country of half a million people finished fourth on a single column because tens of millions of people went looking for him at once.

One blended number would have flattened all of that into a rank. Keeping the three signals visible is what lets you say why a player is on the board, and in Vozinha's case the why is the entire story.

The dashboard is live at Sports Pulse, with the full board, per player pages with the follower and search curves, per-match breakdowns, and a page that walks through the calculation on real data. I would start with Vozinha's page. The curve is the entire reason this exists.

Semantic Image Search with Elasticsearch

Tomás Murúa — Mon, 06 Jul 2026 22:32:52 +0000

Searching Instagram by hashtag or caption text misses the point when the content is visual. A photo of a Samoyed in the snow doesn't always say "Samoyed" in the caption, but you'd still want to find it by typing "a white fluffy dog in the snow." This tutorial builds a pipeline that searches images by what they look like, not what the caption says.

We'll fetch a public profile's feed with SerpApi's Instagram Profile API, embed every cover image into a shared text-image vector space with Jina AI, store those vectors in Elasticsearch, and search with plain language. By the end, you'll have a working search interface where typing "a dog wearing sunglasses" returns the matching image ranked by visual similarity, no caption needed.

Results for the search "a dog wearing sunglasses"

The complete project, notebook and Streamlit app, is on GitHub:

tutorials/python_projects/instagram-semantic-image-search at master · serpapi/tutorials

Prerequisites

Python 3.9+
SerpApi API key (free tier, 250 searches/month)
Jina AI API key (free tier, 10 million tokens)
Elasticsearch cluster (9.0 or higher)
~2 GB disk space for downloaded images

Structured Image Search at Scale

The hard part of image search has never been the vector math. It's getting the images in the first place. Public Instagram profiles show thousands of posts, but there's no "download all" button and no official bulk API from Instagram.

SerpApi Instagram Profile API solves the access problem. One call with engine="instagram_profile" and a username returns the profile's posts along with captions, like counts, comment counts, and a direct URL to each cover image.

That means a few dozen API calls give us hundreds of high-quality images with their metadata, ready to embed.

We'll use the @apple profile as our example. The feed spans landscapes, portraits, animals, and architectural photos, which gives the search plenty of visual variety to work with.

How It Works

Four steps take us from a username to searchable images.

Fetch: Page through the profile's feed with SerpApi, collecting cover images (images + video thumbnails).
Embed: Convert each image into a vector that captures its visual meaning, using Jina AI.
Index: Store the vector plus the metadata in Elasticsearch dense_vector field.
Search: Type a query and get the ranked images by similarity.

The four-step search pipeline

What Are Embeddings?

An embedding is a list of numbers (a vector) that captures what an image means, its visual features, objects, colors, and composition. Similar images produce vectors close together in this high-dimensional space; different images produce vectors far apart.

An image and a sentence become nearby vectors

The key that makes text-to-image search possible is that models like Jina V5 Omni are trained on image-text pairs, so they map both images and text into the same space. A photo of a sunset and the sentence "a person at sunset" end up near each other, not because of keywords but because the model learned their meaning is similar.

Results for the search "a person at sunset"

How Vector Search Works

Once all images are embedded and stored, the search flow is straightforward.

Jina embeds your text query "a dog wearing sunglasses" into a vector using the same model.
Elasticsearch compares that query vector against every stored image vector using cosine similarity, a measure of how close two vectors point in the same direction.
The k nearest neighbors are returned, ranked by similarity score. Scores closer to 1.0 mean more similar; scores closer to 0 mean unrelated.

The query never touches the captions. The ranking is purely visual.

The query vector finds its nearest neighbors

Why These Three Tools?

Component	Role	Why this one
SerpApi	Data access	Structured JSON from public Instagram profiles, paginated, no scraping infrastructure to maintain
Jina v5 Omni	Embedding model	Open-weight, shared text-image space in one model, free API tier
Elasticsearch	Vector store + search	Native `dense_vector` field with built-in kNN, exact and approximate search out of the box, scales from hundreds to millions of vectors

One model for everything: Jina v5 Omni handles both images and text in a single pipeline. If you later want to add semantic search over captions or post metadata, the same model and the same index work. No second model, no separate pipeline.
No raw images stored in the database: We store only the vector, a local file path, and metadata (caption, post URL, likes). Images live in a local images/ folder, just long enough to embed and display results.

What Is Elasticsearch?

If you haven't used Elasticsearch before, it's a search engine (the same technology behind site search on large websites) that also handles vector search natively. You store documents; each document has fields, and you query against those fields. For our use case, one of those fields is a vector, a long list of numbers that represents the visual content of an image.

The relevant feature is vector search via the knn retriever. Given a query vector, it finds the closest stored vectors and returns them ranked by similarity. That's our image search.

Two Ways to Run It

Two options for the Elasticsearch cluster. The notebook code is identical either way. Only ES_URL and ES_API_KEY in .env change.

Local Docker

We start a local single-node cluster using the Elastic start-local script:

curl -fsSL https://elastic.co/start-local | sh

This prints an ES_LOCAL_API_KEY. Put it in your .env as ES_API_KEY. The endpoint is http://localhost:9200.

Elastic Cloud

Create a free project on Elastic Cloud (14-day free trial). Copy the endpoint URL and API key into .env. Good if your machine is light on RAM or you want persistence without managing Docker.

The Pipeline

Step 1. Fetch the Profile

We'll page through the profile feed using the SerpApi Python client:

import serpapi

serp_client = serpapi.Client(api_key=os.getenv("SERPAPI_API_KEY"), timeout=30)

base = {"engine": "instagram_profile", "profile_id": "apple"}
params = dict(base)

for page in range(1, max_pages + 1):
    results = serp_client.search(params)
    profile = results.get("profile_results", {})
    posts = profile.get("posts", [])

    next_token = results.get("serpapi_pagination", {}).get("next_page_token")
    if not next_token:
        break
    params = {**base, "next_page_token": next_token}

Each request returns about 12 posts along with a next_page_token, which we send in the next call to get the next batch. When no token comes back, we've reached the end of the feed. So a few dozen calls are enough to pull hundreds of images.

Step 2. Embed the Images

Here we are sending the images downloaded from the Instagram API to the Jina Embedding API as base64, receiving one vector per image:

JINA_URL = "https://api.jina.ai/v1/embeddings"
JINA_MODEL = "jina-embeddings-v5-omni-small"

def embed_images(images_b64):
    inputs = [{"image": f"data:image/jpeg;base64,{b}"} for b in images_b64]
    payload = {"model": JINA_MODEL, "task": "retrieval.passage", "dimensions": 1024, "input": inputs}
    r = requests.post(JINA_URL, headers=jina_headers, json=payload, timeout=120)
    r.raise_for_status()
    data = sorted(r.json()["data"], key=lambda d: d["index"])
    return [d["embedding"] for d in data]

The one detail worth calling out is task="retrieval.passage". Jina uses different task modes for the vectors you store (retrieval.passage) and the vectors you search with (retrieval.query). Pairing them this way is what Jina recommends for retrieval. Step 4 handles the query side.

Step 3. Create the Index

We create an Elasticsearch index with a vector field to store the embeddings:

es.indices.create(
    index="instagram_photos",
    mappings={"properties": {
        "embedding": {"type": "dense_vector", "dims": 1024, "similarity": "cosine"},
        "caption": {"type": "text"},
        "shortcode": {"type": "keyword"},
        "username": {"type": "keyword"},
        # ... post_url, image_url, is_video, liked_by_count, comments_count
    }}
)

We use cosine similarity because it's the standard match for text and image embeddings, and 1024 dimensions matches Jina's output. See the notebook for the full mapping.

Step 4. Search

We embed the query with Jina (this time with task="retrieval.query") and pass the vector to Elasticsearch's kNN retriever:

def search(query, username, k=6):
    query_vector = embed_query(query)
    resp = es.search(
        index="instagram_photos",
        retriever={"knn": {
            "field": "embedding",
            "query_vector": query_vector,
            "k": k,
            "filter": {"term": {"username": username}},
        }},
        size=k,
        source_excludes=["embedding"],
    )
    return resp["hits"]["hits"]

The filter scopes results in a single profile even when the index holds many. source_excludes keeps the response lean by dropping the 1024-float vector we don't need back.

Results

With 577 images indexed from the Apple profile, the search delivers exactly what you'd expect.

The top result is a video thumbnail (poster frame) from Apple's feed, a dog in sunglasses, scored at 0.737 cosine similarity. No caption matching. No keyword overlap. The pipeline matched the visual content of the image to the meaning of the text query.

A dog wearing sunglasses

Results for the search "a dog wearing sunglasses"

More examples from the same index:

Snow-capped mountains

Results for the search "snow-capped mountains"

Horseback Riding

Results for the search "horseback riding"

Someone holding a phone

Results for the search "someone holding a phone"

A close-up portrait

Results for the search "a close-up portrait"

Beyond Instagram

The embedding and search code doesn't change. Only the SerpApi engine and the fields you extract differ. The same architecture applies to any visual data source SerpApi can reach.

Source	SerpApi engine	Image field	What you'd search
YouTube Search	`youtube`	`thumbnail.static`	Video thumbnails by visual content
Google Images	`google_images`	`original`	Web images matching a topic
Google Shopping	`google_shopping`	`thumbnail`	Product photos
Google Lens	`google_lens`	`thumbnail`	Visual matches for a source image
Amazon Search	`amazon`	`thumbnail`	Product gallery images

Where to Go from Here

Concrete next steps to extend this pipeline:

Hybrid search combining captions and embeddings: Add a text query over the caption field alongside the vector search using Elasticsearch's RRF (Reciprocal Rank Fusion). Posts where both the image and the caption match will rank higher.
Index multiple profiles: The username field already supports multi-profile indexing. Fetch a second profile, embed its images, and use the filter to search within or across profiles.
Try other SerpApi engines from the table above: Swap the fetch step for YouTube thumbnails or Google Shopping product photos. The embedding and search code stays identical.

Conclusion

This pipeline reads images the way people describe them, not the way someone happened to describe or tag them. That single capability changes what you can do with an image catalog.

The Instagram example is just a demonstration. An e-commerce store can surface products by what they look like, not by category tags. A media archive can retrieve photos by scene, mood, or subject. An LLM agent can query the same index in plain language and reason over the matches, no captions or manual tagging required.

Both sides of the search now carry meaning. The query is interpreted by what it means, and the images are stored by what they show. That opens a second layer of interaction between your users, or an LLM, and your content.

How to Build a Brand Monitoring Dashboard with SerpApi and Python

Tomás Murúa — Mon, 22 Jun 2026 13:19:43 +0000

Knowing what people say about your product usually means checking Google News, scrolling through YouTube, and digging into different social media threads. That's three tabs, three interfaces, and no way to compare what you find. This tutorial builds a single dashboard that pulls brand mentions from all three sources using Python and SerpApi.

By the end, you'll have a Streamlit app with three tabs, one for news articles, one for YouTube videos, and one for social media and forum discussions. We'll use "serpapi" as the search query, but you can swap the brand or product name.

Brand monitoring dashboard showing metrics row with total mentions, news articles, YouTube videos, and perspectives counts

Set Up Your Environment

Requirements:

Python 3.8+
SerpApi API Key (the free plan includes 250+ searches/month)
Dependencies (serpapi, pandas, streamlit, altair)

The serpapi package is the official Python SDK. It handles request signing, retries, and response parsing.

The complete code, including a Jupyter notebook version, is available in the SerpApi tutorials repository.

tutorials/python_projects/brand-monitoring-dashboard at master · serpapi/tutorials

The Pipeline

The app follows the same three-step pattern from the GitHub Issues dashboard, fetch raw data, transform it, and display the analysis.

Pipeline diagram showing three stages: fetch, transform, and display

The difference this time is three separate engines running in parallel. Each returns a different response structure, so the transform step normalizes everything into DataFrames before the dashboard consumes it.

Fetch the Data

A single SerpApi client instance works for all three engines:

import serpapi
import os

SERPAPI_KEY = os.environ.get("SERPAPI_KEY", "")
client = serpapi.Client(api_key=SERPAPI_KEY)

Google News

The Google News API returns articles through the news_results key. Each result includes title, link, source (a dict with name and icon), date, and snippet.

def fetch_news(client, brand):
    """Fetch news articles mentioning the brand via Google News."""
    results = client.search({
        "engine": "google_news",
        "q": brand,
        "gl": "us",
        "hl": "en",
    })
    return results.get("news_results", [])

For more use cases with this engine, refer to the news monitoring.

YouTube

The YouTube Search API uses search_query instead of q, and the sp parameter controls time filters. The values EgIIAw%3D%3D (this week) and EgIIBA%3D%3D (this month) are YouTube's internal encoding for upload date filters. You can grab these from YouTube's URL bar after applying a filter manually.

We run both filters and deduplicate by link, since the month results include everything from the week:

YT_FILTER_WEEK = "EgIIAw%3D%3D"
YT_FILTER_MONTH = "EgIIBA%3D%3D"

def fetch_youtube(client, brand):
    """Fetch YouTube videos, combining week and month filters."""
    seen = set()
    videos = []

    for sp_filter in (YT_FILTER_WEEK, YT_FILTER_MONTH):
        results = client.search({
            "engine": "youtube",
            "search_query": brand,
            "sp": sp_filter,
        })
        for video in results.get("video_results", []):
            link = video.get("link", "")
            if link and link not in seen:
                seen.add(link)
                videos.append(video)

    return videos

For more examples using the YouTube API, refer to this link.

Google Perspectives

Google Perspectives API surfaces user-generated content from LinkedIn, Reddit, Quora, and blogs. It uses the standard Google engine, and the results appear under the perspectives key:

SerpApi search with the perspectives result

def fetch_perspectives(client, brand):
    """Fetch user-generated content (Reddit, LinkedIn, Quora)."""
    results = client.search({
        "engine": "google",
        "q": brand,
        "google_domain": "google.com",
    })
    return results.get("perspectives", [])

Fetch in Parallel

Three sequential API calls take roughly three seconds. Running them in parallel with Python ThreadPoolExecutor brings that down to about one second. Each call runs in its own thread while the others wait for their response:

from concurrent.futures import ThreadPoolExecutor

@st.cache_data(ttl=300)
def fetch_all_mentions(brand):
    """Fetch all brand mentions from three engines in parallel."""
    client = serpapi.Client(api_key=SERPAPI_KEY)

    with ThreadPoolExecutor(max_workers=3) as pool:
        news_future = pool.submit(fetch_news, client, brand)
        yt_future = pool.submit(fetch_youtube, client, brand)
        persp_future = pool.submit(fetch_perspectives, client, brand)
        return news_future.result(), yt_future.result(), persp_future.result()

SerpApi also offers a server-side async parameter for large-scale batch processing, where you submit searches and retrieve results later. For our three concurrent calls, client-side threading is simpler and equally effective.

The @st.cache_data(ttl=300) decorator caches results for 5 minutes. Without it, every Streamlit interaction would re-trigger the API calls. This works alongside SerpApi's own 1-hour result cache, which serves identical queries from the cache at no extra search cost unless you explicitly pass no_cache=true. Together, these two layers minimize redundant API calls during development and testing.

For more optimization techniques when working with SerpApi at scale, refer to this blog.

Transform the Data

All three engines return dates as relative strings ("3 hours ago", "2 days ago"). We need a shared parser to convert them into datetime objects for sorting.

Parse Relative Dates

Two details worth noting. The regex is compiled once and reused since this function runs for every result in all three engines. And the fallback returns datetime.now(timezone.utc) instead of None, so results without a parseable date sort to the top rather than breaking pandas operations.

import re
from datetime import datetime, timedelta, timezone

RELATIVE_DATE_RE = re.compile(
    r"(\d+)\s+(second|minute|hour|day|week|month|year)s?\s+ago", re.IGNORECASE
)

UNIT_TO_TIMEDELTA = {
    "second": lambda n: timedelta(seconds=n),
    "minute": lambda n: timedelta(minutes=n),
    "hour": lambda n: timedelta(hours=n),
    "day": lambda n: timedelta(days=n),
    "week": lambda n: timedelta(weeks=n),
    "month": lambda n: timedelta(days=n * 30),
    "year": lambda n: timedelta(days=n * 365),
}

def parse_relative_date(text):
    """Convert '3 hours ago' into a datetime object."""
    if not text:
        return datetime.now(timezone.utc)

    match = RELATIVE_DATE_RE.search(str(text))
    if not match:
        return datetime.now(timezone.utc)

    amount = int(match.group(1))
    unit = match.group(2).lower()
    delta = UNIT_TO_TIMEDELTA.get(unit, lambda n: timedelta())(amount)
    return datetime.now(timezone.utc) - delta

Build DataFrames

Each engine gets into its own transformer. Here's the news version:

def transform_news(results):
    """Convert raw Google News results into structured records."""
    records = []
    for item in results:
        source = item.get("source") or {}
        source_name = source.get("name", "Unknown") if isinstance(source, dict) else str(source)

        records.append({
            "title": item.get("title", ""),
            "link": item.get("link", ""),
            "source": source_name,
            "date": parse_relative_date(item.get("date", "")),
            "snippet": item.get("snippet", ""),
        })
    return records

The source field can be a dict or a plain string depending on the result, so the isinstace check handles both. YouTube and Perspectives follow the same pattern, with two differences worth highlighting.

YouTube views come back as strings like "1,234 views", so we strip non-numeric characters before converting:

views = item.get("views") or 0
if isinstance(views, str):
    views = int(re.sub(r"[^\d]", "", views) or 0)

Build the Dashboard

The Streamlit interface starts with a form for the brand query and a row of summary metrics across all three sources:

st.set_page_config(page_title="Brand Monitoring Dashboard", layout="wide")
st.title("Brand Monitoring Dashboard")

with st.form("brand_form"):
    brand = st.text_input("Brand or keyword to monitor", value="serpapi")
    submitted = st.form_submit_button("Search")

Brand or keyword selector to monitor

After fetching, the dashboard shows four metrics at the top for a quick overview, then splits into three tabs:

col1, col2, col3, col4 = st.columns(4)
col1.metric("Total Mentions", total_mentions)
col2.metric("News Articles", len(news_records))
col3.metric("YouTube Videos", len(yt_records))
col4.metric("Perspectives", len(persp_records))

Dashboard metrics row displaying total mentions across three sources

News Tab

The News tab pairs an Altair bar chart of top sources with a sortable table. Altair ships with Streamlit, so there's nothing extra to install. We use it instead of st.bar_chart because it gives control over orientation, tooltips, and styling.

source_df = news_df["source"].value_counts().head(10).reset_index()
source_df.columns = ["source", "count"]

source_chart = alt.Chart(source_df).mark_bar(
    cornerRadiusTopRight=4, cornerRadiusBottomRight=4
).encode(
    x=alt.X("count:Q", title="Articles"),
    y=alt.Y("source:N", sort="-x", title=""),
    color=alt.value("#4A90D9"),
    tooltip=["source:N", "count:Q"],
).properties(height=350)
st.altair_chart(source_chart, use_container_width=True)

News tab with horizontal bar chart of top sources and sortable article table

The table uses st.column_config.LinkColumn so each article title links directly to its source.

YouTube Tab

The YouTube tab shows views by channel and a sorted video table. The chart groups views by channel to surface which creators talk about the brand the most.

channel_df = yt_df.groupby("channel")["views"].sum().reset_index()
channel_df = channel_df.sort_values("views", ascending=False).head(10)

channel_chart = alt.Chart(channel_df).mark_bar(
    cornerRadiusTopRight=4, cornerRadiusBottomRight=4
).encode(
    x=alt.X("views:Q", title="Views", axis=alt.Axis(format="~s")),
    y=alt.Y("channel:N", sort="-x", title=""),
    color=alt.value("#4A90D9"),
    tooltip=["channel:N", alt.Tooltip("views:Q", format=",")],
).properties(height=350)

YouTube tab showing views by channel chart and video table

Perspectives Tab

The Perspectives tab splits the layout between a discussion table on the left, and a donut chart of mentions by platform on the right. The donut chart makes it easy to see where conversations happen, whether it's LinkedIn, Reddit, X, etc.

platform_chart = alt.Chart(platform_df).mark_arc(
    innerRadius=60, outerRadius=120
).encode(
    theta=alt.Theta("count:Q"),
    color=alt.Color("source:N", legend=alt.Legend(title="Platform")),
    tooltip=["source:N", "count:Q"],
).properties(height=350)

Perspectives tab with discussions table on the left and donut chart of mentions by platform on the right

When to use this approach

Ideal for:

Tracking brand mentions across news, video, and social in one view
Monitoring product launches, PR campaigns, or competitor names
Building internal dashboard for marketing or DevRel teams

Not recommended for:

Real-time alerting. The API returns a snapshot, not a stream. For notifications, schedule the script on an interval and compare results.
Historical analysis. Each engine returns recent results, not a complete archive.

If you want to explore the API response before writing code, the SerpApi Playground lets you test any engine interactively. And if you only need news coverage, the Google News API alone handles most brand monitoring use cases.

Where to Go from Here

This dashboard gives you a live snapshot. The natural next step is turning it into a historical record. Store each fetch in a database (SQLite, PostgreSQL, or even a CSV) and you can compare mention volume week over week, track which sources cover your brand consistently, and spot trends that a single snapshot can't show.

With historical data in place, you can layer on more analysis. Identify content gaps by looking at what topics competitors get covered on but you don't. Track which YouTube channels mention your product and how their view counts trend over time. Flag new platforms or authors that start discussing your brand.

The data is yours to work with however fits your needs. The three engines give you the raw material; what you build on top depends on the questions you're trying to answer.

Conclusion

The full application is about 350 lines in a single Python file. Three API calls, three DataFrames, three tabs. The query input at the top lets you switch brands without changing the code.

What started as a way to check where "serpapi" shows up on the web became a tool that surfaces patterns you miss manually. The Perspectives tab pulls in LinkedIn posts, Reddit threads, and Quora answers that don't appear in regular news or video searches, and combining them in one view gives you the full picture.

How to Analyze Issues with the GitHub API and Python

Tomás Murúa — Fri, 19 Jun 2026 13:50:37 +0000

If you maintain an open-source project or track bugs in a public repository, you know how fast GitHub issues can pile up. This tutorial shows you how to pull issue data from the GitHub REST API, transform it into structured metrics, and visualize trends.

By the end, you'll have a Streamlit dashboard that shows open vs closed counts, issue aging, service breakdown, and trends over time. The entire app is a single Python file; I built it against the SerpApi Public Roadmap repository, which has over 3,600 issues, but it works with any public GitHub repo.

Streamlit dashboard analyzing 1,900+ GitHub issues from serpapi/public-roadmap

Set Up Your Environment

We'll use Streamlit, a Python framework that turns scripts into interactive web apps with minimal code. You need:

Python 3.8+
GitHub personal access token
Dependencies (pandas, streamlit, requests)

GitHub recommends the fine-grained tokens over the classic token. Create one with "public repositories" selected under repository access.

Export your GitHub token and run the app:

export GITHUB_TOKEN="your_token_here"
streamlit run app.py

You can find the complete app and notebook in the GitHub repository. The tutorial below walks through each piece of the code.

tutorials/python_projects/github-issues-dashboard at master · serpapi/tutorials

The Pipeline

The app follows three steps:

Fetch the raw issues from GitHub.
Transform them into structured data.
Display the analysis.

Fetch-Transform-Analyze pipeline for GitHub issue data

How to Fetch the Issues

The GitHub REST API exposes issues through GET /repos/{owner}/{repo}/issues. Each request returns up to 100 issues, so repositories with more require pagination.

Here's the fetch function. Two things to note in the script:

The pull_request check filters out PRs (GitHub's Issues endpoint returns both).
The 422 check handles a pagination limit we'll explain below.

def fetch_issues_by_state(owner, repo, state, headers):
    """Fetch all issues for a given state (open/closed), handling pagination."""
    issues = []
    page = 1

    while True:
        response = requests.get(
            f"{BASE_URL}/repos/{owner}/{repo}/issues",
            headers=headers,
            params={"state": state, "per_page": 100, "page": page},
        )

        # GitHub returns 422 when pagination exceeds ~1,000 results
        if response.status_code == 422:
            break

        # Rate limit exceeded
        if response.status_code in (403, 429):
            raise Exception(
                "GitHub API rate limit exceeded. "
                "Wait a few minutes and try again, or check your GITHUB_TOKEN."
            )

        response.raise_for_status()
        data = response.json()

        if not data:
            break

        for issue in data:
            # The Issues endpoint returns PRs too; filter them out
            if "pull_request" not in issue:
                issues.append(issue)

        page += 1

    return issues

Pagination Limit

When I first tested this script, I used state=all to fetch everything in one loop. It worked until page 11, when GitHub returned a 422 Unprocessable Entity. GitHub cap pagination at roughly 1,000 results per query. Page 11 at 100 per page exceeds that limit.

Each query stays under the 1,000-result cap. This gives us roughly 1,900 issues total, enough for a meaningful analysis.

The @st.cache_data(ttl=300) decorator is specific to Streamlit. It caches the return value for 5 minutes. Without it, Streamlit re-executes the entire script on every user interaction. A single click would trigger 20+ API requests.

The SerpApi public roadmap repository has over 3,600 issues. A single query can't reach them all. The fix is to split into two separate queries:

@st.cache_data(ttl=300)
def fetch_all_issues(owner, repo):
    """Fetch all issues from the repo, excluding pull requests."""
    headers = {}
    if GITHUB_TOKEN:
        headers["Authorization"] = f"Bearer {GITHUB_TOKEN}"

    open_issues = fetch_issues_by_state(owner, repo, "open", headers)
    closed_issues = fetch_issues_by_state(owner, repo, "closed", headers)

    return open_issues + closed_issues

Rate limits: without a token, you get 60 requests per hour. With a personal access token, 5,000. For this app, always authenticate. For more details, refer to the rate limit docs.

Transform the Issues

The raw API response has dozens of fields per issue. Most are irrelevant for analysis. The transformation extracts what matters and processes the derived metrics.

Here's what a single issue looks like when we fetch it from the API:

{
    "number": 3654,
    "title": "[Google search API] - shows only images, FAQ, AI snippet or video on a valid
  search (Location error)",
    "state": "closed",
    "created_at": "2026-03-26T14:03:30Z",
    "closed_at": "2026-03-26T14:50:58Z",
    "labels": [
      {
        "name": "type: bug",
        "color": "FEF2C0",
        "description": "Something is broken"
      }
    ],
    "user": {
      "login": "annieiscorrie"
    },
    "comments": 4,
    "html_url": "https://github.com/serpapi/public-roadmap/issues/3654",
    "pull_request": null
  }

The basic fields (number, title, state, created_at) come straight from the API response. The interesting part is these specific fields:

age_days: days since creation. This is the metric that surfaces forgotten issues. Sorting by age immediately shows what's been sitting unresolved the longest.
service: the serpapi/public-roadmap repository encodes the affected component in the title using square brackets, like [Google Search API] Results missing pagination becomes "Google Search API". The regex \[(.+?)\] captures the text inside the first pair of brackets.

Not every repository uses brackets in titles, the other commons alternatives include label prefixes area/component/, milestones, or GitHub projects. Adjust the services extraction in transform_issue to match your repo's conventions.

status and type: parsed from labels with status: and type: prefixes. Check your target repository's labels before assuming a format.

def transform_issue(issue):
    """Convert a raw GitHub issue dict into a structured record."""
    created = datetime.fromisoformat(issue["created_at"].replace("Z", "+00:00"))
    age_days = (datetime.now(timezone.utc) - created).days
    labels = [label["name"] for label in issue.get("labels", [])]

    # Extract service from title prefix like "[Google Search]"
    match = re.search(r"\[(.+?)\]", issue["title"])
    service = match.group(1) if match else "General"

    # Extract status and type from label prefixes
    status = next(
        (l.split(": ", 1)[1] for l in labels if l.startswith("status:")), "none"
    )
    type_ = next(
        (l.split(": ", 1)[1] for l in labels if l.startswith("type:")), "none"
    )

    return {
        "number": issue["number"],
        "title": issue["title"],
        "state": issue["state"],
        "created_at": created.strftime("%Y-%m-%d"),
        "age_days": age_days,
        "labels": labels,
        "service": service,
        "status": status,
        "type": type_,
    }

Analyze the Issues

With the data transformed into a pandas DataFrame, Streamlit handles the display. The dashboard starts with a repository selector so you can analyze any public repo without changing the code:

with st.form("repo_form"):
    col_owner, col_repo, col_btn = st.columns([2, 2, 1])
    with col_owner:
        owner = st.text_input("Owner", value="serpapi")
    with col_repo:
        repo = st.text_input("Repository", value="public-roadmap")
    with col_btn:
        submitted = st.form_submit_button("Fetch Issues")

Repository selector for switching between public GitHub repos

After submitting, the dashboard displays four sections.

Display a Metrics Summary

Three numbers at the top for a quick overview:

col1, col2, col3 = st.columns(3)
col1.metric("Total Issues", total)
col2.metric("Open", open_count)
col3.metric("Closed", closed_count)

Summary metrics: total, open, and closed issue counts

Chart Status and Type Distribution

A bar chart for statuses and a donut for types side by side. We use Altair instead of Streamlit's built-in st.bar_chart because it gives control over chart type, orientation, tooltips, and styling. Altair ships with Streamlit, so there's nothing extra to install.

col_left, col_right = st.columns(2)

with col_left:
    st.subheader("Issues by Status")
    status_df = df["status"].value_counts().reset_index()
    status_df.columns = ["status", "count"]

    status_chart = (
        alt.Chart(status_df)
        .mark_bar(cornerRadiusTopLeft=4, cornerRadiusTopRight=4)
        .encode(
            x=alt.X("status:N", sort="-y", title=""),
            y=alt.Y("count:Q", title="Issues"),
            color=alt.value("#4A90D9"),
        )
        .properties(height=350)
    )
    st.altair_chart(status_chart, use_container_width=True)

Issues by status (bar chart) and by type (donut chart) using Altair

Track Issues Over Time

An area chart showing issues opened per month, split by state:

df["month"] = pd.to_datetime(df["created_at"]).dt.to_period("M").astype(str)
monthly_state = df.groupby(["month", "state"]).size().reset_index(name="count")

timeline = (
    alt.Chart(monthly_state)
    .mark_area(opacity=0.6)
    .encode(
        x=alt.X("month:N", title=""),
        y=alt.Y("count:Q", title="Issues"),
        color=alt.Color("state:N"),
        tooltip=["month:N", "state:N", "count:Q"],
    )
    .properties(height=350)
)
st.altair_chart(timeline, use_container_width=True)

Monthly issue activity split by open and closed state

Compare Services and Measure Aging

The last row pairs two charts, one with services that have the most issues and the other with how old the open ones are. And for aging, we bucket open issues into time ranges:

def age_bucket(days):
    if days < 7:
        return "< 7 days"
    elif days < 30:
        return "7-30 days"
    elif days < 90:
        return "30-90 days"
    elif days < 180:
        return "90-180 days"
    elif days < 365:
        return "180-365 days"
    else:
        return "> 365 days"

Top 15 services by issue count and open issue aging distribution

Dashboard Complete View

Here's the full dashboard running against serpapi/public-roadmap. The repository selector at the top lets you switch to any public repo. All five sections update together when you fetch a new repository.

Complete dashboard view running against serpapi/public-roadmap

When to Use This Approach

Ideal for:

Track the health of open-source projects you depend on or maintain.
Building internal dashboards for product roadmap visibility.

Not recommended for:

Repositories with fewer than 50 issues, because the manual review will be faster.
Real-time monitoring and the API rate limits make the polling impractical.
Fetching complete history from repositories with 10,000+ issues.

Conclusion

The full application is 220 lines in a single Python file. The dashboard includes a repository selector, so you can point it at any public GitHub repository without touching any code, and everything runs locally, so you have control over everything.

What started as a quick experiment to understand a public roadmap became a tool I use daily. The GitHub API gives everything you need; the work is in structuring the data to answer the question that the web interface doesn't provide.

How to Connect Your Local LLM with Web Search Data

Tomás Murúa — Thu, 04 Jun 2026 05:48:28 +0000

This blog will show you how to give a local LLM access to real-time web data. We'll build a small chat app in TypeScript; the model will decide what to search for, your code will run the search, and the model will write an answer with the fresh data in hand. The inference runs on your hardware, and the only thing that leaves your local environment is the search query the model chose to issue.

By the end of this blog, you'll understand the full request flow, you'll know how to keep a large search response from blowing past your context window, and you'll have a public repository you can clone and run against your own local model.

Why LLMs Don't Have Access to Real-Time Data

Every large language model is limited by its training data cutoff. Anything that happened after that date is invisible to the model. That's true whether you run the model locally or call a hosted API.

Cloud chat products hide this from most users. ChatGPT, Claude, and Gemini ship the model wrapped in a product, and that product includes a built-in search tool. When you ask ChatGPT, "What's the stock price of Nvidia right now?" the model doesn't answer on its own; the product called a search tool, fed the result back to the model, and the model wrote the up-to-date reply.

The moment you skip that product and run a model locally, none of it is wired up. You get the model on its own; no tools, no search, no real-time data. The fix is what the big AI labs already do under the hood. Give the model a tool and let it decide when to call it.

The tool, in our case, is a web search API; one API call returns structured search results instead of raw HTML. We'll use SerpApi, which returns clean JSON across Google Search, Finance, News, Maps, Flights, and 100+ other search engines.

Run a Local Model with LM Studio

There are different ways to use LLMs locally, but in this guide, we're going to use LM Studio. This is a desktop app for downloading and running open-weight models on your own hardware. It matters for this project for one reason; it exposes the model through OpenAI and Anthropic standards on localhost. You point the SDK you already use at that endpoint, and the same code that talks to GPT or Claude now talks to your local model.

Here's how to get a model running locally in a few minutes:

Download and install LM Studio from lmstudio.ai.
Open the model browser and download a model with native tool calling. I'm using Qwen3.5-9B, which runs comfortably on an M4 Max with 36 GB of unified memory.

Open the Developer tab, load the model, and click Start Server.

LM Studio now serves an OpenAI-compatible endpoint at http://localhost:1234/v1.

The server has to stay running for the rest of this guide; that local endpoint is what our code talks to. It is not necessary to pre-load the model because the app requests it on demand, so LM Studio loads it only when a call actually comes in.

Any model with tool calling works here, so favor one built for agentic workflows; that realibility is what the whole loop leans on. Google's Gemma 4 is another strong option. Which mode and size you pick comes down to your hardware; it has to fit in memory alongside a context window large enough to hold the search result.
If you're not sure what your hardware can handle, LM Studio's model search flags the best match for your hardware and warns you when a model is too large to run. I chose Qwen3.5-9B because it offered a good balance between speed and the quality of its responses.

Build the Chat App

We'll build a chat app that wraps a local model in the same tool-calling loop that the cloud products use. The loop is the back-and-forth between the model and your code: the model asks for a search, your code runs it, and the model writes the answer.

The full project is a TypeScript app you can clone from:

GitHub Repository

Currently, it supports five SerpApi engines (Search, Finance, News, Maps, and Flights), and you can add more. The architecture has three moving parts:

The third part is the one most tutorials skip, and it's the one that makes a local setup actually work: a search response is far too big to fit in a model with a small context window. We'll fix that with a single SerpApi parameter called JSON Restrictor.

Set Up the Project

Clone the repo, install dependencies, and grab an API key from your SerpApi dashboard:

git clone https://github.com/serpapi/local-llm-web-search.git
cd local-llm-web-search
bun install
echo "SERPAPI_API_KEY=your_key_here" > .env

Make sure that the server is running in LM Studio, then start the app.

bun dev

With the local server running and the key in place, you're ready to walk through how the loop works.

How the Loop Works

The pattern is the one OpenAI documents for function calling and Anthropic documents for tool use. Five steps, in order. The code below is lifted from the app's agent.ts and restrictors.ts, trimmed to the essentials.

Step 1. Declare the Tools to the Model

You hand the model a list of tools it can call. Each tool has a name, a description, and a JSON Schema for its parameters. Here is an example of declaring SerpApi as the web search tool the model can reach for:

import OpenAI from "openai"

const TOOLS: OpenAI.Chat.Completions.ChatCompletionTool[] = [
  {
    type: "function",
    function: {
      name: "google_search",
      description:
        "Search Google for current web results. Use for news, facts, or anything time-sensitive.",
      parameters: {
        type: "object",
        properties: {
          query: { type: "string", description: "What to search for" },
          gl: { type: "string", description: "Two-letter country code, e.g. 'us'" },
          hl: { type: "string", description: "Two-letter language code, e.g. 'en'" },
        },
        required: ["query"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "google_flights_search",
      description: "Find flight options between two airports for a given date.",
      parameters: {
        type: "object",
        properties: {
          departure_id: { type: "string", description: "IATA code, e.g. 'JFK'" },
          arrival_id: { type: "string", description: "IATA code, e.g. 'LAX'" },
          outbound_date: { type: "string", description: "YYYY-MM-DD" },
          return_date: { type: "string", description: "YYYY-MM-DD, round-trip" },
        },
        required: ["departure_id", "arrival_id", "outbound_date"],
      },
    },
  },
  // ...one entry per SerpApi engine you want to support
]

A few things worth noting:

The descriptions matter more than they look; the model picks tools by reading them, so write each one like documentation for a developer who has never seen your code before.
Each tool name maps to one SerpApi engine. The app supports google_search, google_finance_search, google_news_search, google_maps_search, and google_flights_search.
These definitions go to the model on every request, so keep them lean. They count against the same context window budget as everything else.

Step 2. Ask the Model What It Needs

You send the user's prompt to the model along with the tools list. The model returns one of two things:

A direct answer when it already knows
Tools calls when it needs the web data

Point the OpenAI client at the LM Studio server you started earlier:

const client = new OpenAI({
  baseURL: "http://localhost:1234/v1", // LM Studio's OpenAI-compatible endpoint
  apiKey: "lm-studio",                 // any non-empty string works locally
})

const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
  { role: "system", content: "You are an assistant with access to web-search tools." },
  { role: "user", content: userPrompt },
]

const first = await client.chat.completions.create({
  model,
  messages,
  tools: TOOLS,
  tool_choice: "auto",
  temperature: 0.1,
})

A few things worth noting:

The LM Studio server must be running in the background for this call to connect; the model itself loads on demand when the request arrives. If you get a connection refused error, that's the first thing to check.
tool_choice: "auto" lets the model decide between answering directly and calling tools. A low temperature (0.1) keeps tool selection deterministic; we'll raise it for the second call when the model writes the answer.
If first.choices[0].message.tool_calls is non-empty, the model wants the web, and you have work to do.
Before this call, the app trims the conversation history with a sliding window, dropping the oldest turns so a long chat doesn't overflow the budget. json_restrictor keeps each response small; the window keeps the cumulative history small.

Step 3. Run the Tool Calls

For each tool call, validate the arguments and run the matching SerpApi engine. The model can pick more than one tool in the same turn. For example, "Compare Apple and Tesla stock" is going to trigger two finance lookups, so run them concurrently:

const toolCalls = first.choices[0].message.tool_calls ?? []

const results = await Promise.all(
  toolCalls.map(async (call) => {
    const args = validate(call.function.name, JSON.parse(call.function.arguments))
    const data = await executeTool(call.function.name, args)
    return { call_id: call.id, data }
  }),
)

A few things worth noting:

validate runs the arguments through a Zod schema before any network call. Smaller models occasionally emit malformed JSON in arguments, and the check catches it before you spend a SerpApi search on garbage input.
executeTool is where the response gets trimmed. That's the next step, and it's the part that makes the whole thing viable on a local model.

Step 4. Trim the Response with the JSON Restrictor

SerpApi returns everything an engine knows. A single Google Flights response can return over 14,000 tokens once airport metadata, every price tier, layover details, carbon estimates, and baggage policies come along. A Google Finance response can top 46,000 tokens. A Qwen3.5-9B running with a 32K context window can't carry that, especially once you add the system prompt, the conversation history, and the tool definitions themselves.

SerpApi solves this for you with the JSON Restrictor parameter. You pass a field-selector string with the request, and SerpApi filters the response on its servers before sending it back. The trimming happens before the data crosses the network, so the giant payload never reaches your machine at all. Applying it is one line:

import { getJson } from "serpapi"

// Run a tool's SerpApi call with its server-side restrictor applied.
function executeTool(name: RestrictorTool, args: ToolArgs): Promise<SerpApiJson> {
  return getJson({ ...toolParams(name, args), json_restrictor: RESTRICTORS[name] })
}

The restrictor strings live in their own file, one per engine. The syntax is jq-like: .field selects an object field, [] selects every array item, [0:5] takes a slice, and .{a, b} projects multiple fields at once:

export const RESTRICTORS: Record<RestrictorTool, string> = {
  google_search: [
    "answer_box.{answer,snippet,title}",
    "knowledge_graph.{title,type,description,source.{name,link}}",
    "organic_results[0:5].{title,link,displayed_link,snippet,date}",
  ].join(","),

  google_flights_search: [
    "best_flights[0:3].{price,total_duration,type,flights[].{airline,departure_airport.{id,time},arrival_airport.{id,time},duration},layovers[].{name,duration}}",
    "other_flights[0:3].{price,total_duration,type,flights[].{airline,departure_airport.{id,time},arrival_airport.{id,time},duration}}",
  ].join(","),

  // ...one restrictor per engine
}

A few things worth noting:

Fields absent from a given response are silently skipped, so one restrictor can list every branch it might want (an answer_box for factual queries, a knowledge_graph for entities, organic_results as the fallback) without any conditional logic.
json_restrictor works on every SerpApi engine, so the pattern is the same whether you're filtering flights, finance, or maps.
No client-side formatting is needed; the response arrives already shaped for the model.

Here's an example of the before and after from the prompt "What's the current price of NVDA?":

The token reduction is the whole point. These are measured on real queries from the app's benchmark:

Step 5. Send the Second Call and Get the Answer

Push the restricted results back into the conversation and send a second call. The model writes the final answer using the trimmed data:

messages.push(first.choices[0].message) // the assistant's tool_calls turn
for (const { call_id, data } of results) {
  messages.push({
    role: "tool",
    tool_call_id: call_id,
    content: JSON.stringify(data),
  })
}

const second = await client.chat.completions.create({
  model,
  messages,
  temperature: 0.3,
})

return second.choices[0].message.content

A few things worth noting:

Send the assistant's original tool-call message before the role: "tool" results, in that order. The model needs the call context to bind each result back to its tool_call_id.
Temperature climbs from 0.1 to 0.3 here; this is a prose-writing step, not a tool-selection step.
Every tool result must be present before the model can answer. Miss one and the second call fails.

A Full Run: Flights from JFK to LAX

Here's the loop end-to-end with a single prompt:

The model parses the question, picks google_flights_search, and emits one tool call with departure_id: "JFK", arrival_id: "LAX", and a date. The app validates the arguments, then calls SerpApi with the flights restrictor attached. SerpApi returns roughly 800 tokens of pre-filtered JSON instead of the 14,000+ it would otherwise send. The app pushes that into the message history and sends the second call.

The model writes a three-itinerary comparison with airline, price, schedule, duration, and stops. On an M4 Max running Qwen3.5-9B, the full turn (pick tool, fetch, write) lands in a handful of seconds, with the SerpApi call completing well inside the inference time.

One Catch: Open Models Sometimes Emit Tool Calls as Text

The OpenAI spec defines a structured tool_calls array on the model's response. Qwen3.5, gpt-oss, and several other open models will sometimes ignore the array and emit tool calls inline as <tool_call>...</tool_call> text inside message.content instead. Your code has to parse both shapes.

The fix is a fallback that runs whenever the structured array comes back empty:

function parseInlineToolCalls(content: string) {
  const calls = []
  for (const match of content.matchAll(/<tool_call>\s*([\s\S]*?)\s*<\/tool_call>/gi)) {
    try {
      calls.push(JSON.parse(match[1]))
    } catch {
      // skip malformed blocks rather than fail the whole turn
    }
  }
  return calls
}

A few things worth noting:

Run the fallback only when the structured tool_calls array is empty. If both are empty, the model genuinely answered without a tool.
Swallow parse errors per block; one malformed tag shouldn't kill the turn.
Synthesized calls need a `tool_call_id (any unique string works) so you can map results back when you push them into history.

Why Not Use The SerpApi MCP Server?

A reasonable question, since SerpApi also ships an MCP server. Both paths hit the same endpoints; the difference is who drives the loop.

The MCP server is the right transport when the user is the consumer (Claude Desktop, Cursor, or any MCP-compatible client) and the model has a large context window to spare. The client connects to the server and the model calls the tools through it; you don't write the loop yourself.

The direct API path in this guide is the right shape when you're building the agent yourself and your runtime has a tight context budget, which is exactly the case for a local model. You own the loop, so you decide what reaches the model per turn, and json_restrictor lets you make that decision per field. On a 32K context window, that control is the difference between an answer and an overflow.

Where to Go from Here

In this blog we connected a local model to live web data through a five-step loop: declare the tools, ask the model what it needs, run the searches, trim each response with json_restrictor, and send a second call for the answer. The restrictor is doing the quiet, heavy work; without it, a single flights query overflows the context window of a 9B model.

The same loop scales to any SerpApi engine. Add one tool definition and one restrictor string per engine, and the rest of the code stays exactly as it is. Clone the repo, point it at your own local model, and try swapping in an engine the app doesn't cover yet; the shape will already be familiar.