DEV Community: Hilman Ramadhan

SerpApi vs. Firecrawl: Search Intelligence VS Website Content

Hilman Ramadhan — Mon, 27 Jul 2026 11:17:52 +0000

When comparing SerpApi vs Firecrawl, you're not comparing two solutions to the same problem. You're comparing two fundamentally different data extraction philosophies that happen to both involve web scraping.

Firecrawl is a website content scraper. You point it at a URL, techcrunch.com/article/some-news and it returns clean, LLM-ready markdown of that article. It crawls links, extracts text, converts HTML to structured formats, and handles JavaScript rendering. It's designed for AI applications that need to ingest website content for RAG systems, knowledge bases, or training data.

SerpApi is a search engine scraper. You query it with "best CRM software" and it returns what Google (and other top search engines) shows for that query: the top 10 organic results, the paid ads, the featured snippet, the "People Also Ask" box, and the local map pack. It doesn't scrape the content of those websites. It scrapes the search results themselves, the metadata, rankings, prices, ratings, and visibility signals that search engines display.

Firecrawl vs SerpApi

For generic content extraction like "Get me all blog posts from this competitor's site," Firecrawl works fine and is purpose-built for that task. But for business intelligence questions like "Who's ranking for this keyword?" or "What's the average price for this product across retailers?", Firecrawl is categorically the wrong tool.

The core issue isn't that Firecrawl is bad at what it does. It's that Firecrawl and SerpApi solve opposite problems. If your business depends on understanding search visibility, competitive positioning, or what consumers actually see when they search, then scraping individual websites doesn't give you that data. You need to scrape the search engines themselves.

The Search Intelligence Gap: What Firecrawl Cannot Do

While Firecrawl recently added a /search endpoint that can perform basic web searches—including basic location customization—it fundamentally differs from true SERP scraping. Search engine results change drastically depending on the user. SerpApi provides an extensive range of parameters (like granular geolocation, device type, and language settings) to fetch fully customized results exactly as your target audience sees them.

Furthermore, Firecrawl's search returns only simple result metadata like URLs, titles, and descriptions. It cannot extract the structured SERP features that drive business intelligence: paid ad blocks, featured snippets as structured objects, Knowledge Graph panels, People Also Ask sections, local pack results, shopping carousels, or any Google-specific SERP elements.

As Firecrawl's own documentation clarifies: "SERP APIs specifically scrape and reformat data from existing search engines like Google or Bing. All SERP APIs are Search APIs, but not all Search APIs are SERP APIs."

This eliminates entire categories of use cases:

SEO rank tracking with SERP features: While Firecrawl provides position numbers in search results, they don't extract structured SERP features (Featured Snippets, People Also Ask, Knowledge Graphs) as separate objects for comprehensive rank tracking
Local SEO analysis: You cannot extract Google Maps data, local pack rankings with review counts, or business verification status
Price comparison: You cannot scrape Google Shopping SERP features or compare prices across retailers as displayed in search

If your product roadmap includes any feature that starts with "Show me what appears when I search for," Firecrawl is not the tool. SerpApi is.

Read moe: SerpApi vs other Web Search API alternatives.

The Structured Data Problem

Even with Firecrawl's basic search capability, you get raw result lists without the rich structured data that powers business applications.

SerpApi returns JSON with specific parsers for each SERP feature. When you query "pizza near me", SerpApi's response includes structured fields for rating, review counts, position ranking, hours of operation, and price indicators. Firecrawl returns basic fields like title, URL, and description.

Example when querying "Pizza Near me"

You'd have to build your own parser for these SERP-specific data points, which is exactly what SerpApi has industrialized across many platforms including Google, Bing, YouTube, Amazon, with 100+ API endpoints.

Website Scraping vs. Search Scraping: A Concrete Example

Here's a real-world scenario that clarifies the distinction:

Use Case: You're building a competitive intelligence dashboard for e-commerce brands. Your users want to know:

Which competitors rank for their target keywords
What prices competitors show in Google Shopping
Which products have featured snippets or rich results
What ad copy competitors use in paid search

Using Firecrawl:

You could scrape your competitors' product pages to extract their prices and descriptions. But you have no way to know which competitors to scrape because you don't know who ranks for your keywords. You'd need a pre-existing list of competitor URLs. Even with that list, you wouldn't see their Google Shopping prices (which often differ from their website prices), their ad copy, or their SERP rankings.

Using SerpApi:

You query SerpApi's Google Shopping API with your target product keyword. You get back every retailer showing prices for that product in Google Shopping, their exact prices, their ratings, their shipping costs, and their position in the results. You access ad data to see which competitors are bidding on that keyword and what ad copy they're using. All structured data, one API call per query.

The first approach gives you website content. The second gives you market intelligence.

The "LLM-Ready" Misconception

Firecrawl markets itself heavily on returning "LLM-ready markdown." That's true, and for content ingestion use cases, it's valuable. If you're building a chatbot that needs to answer questions about your company's documentation, Firecrawl can crawl your docs and return clean markdown.

But "LLM-ready" doesn't mean "search-intelligence-ready." If your LLM-powered application needs to answer questions like "What are the top 5 CRM tools according to Google?" or "Which hotels in Miami are under $200 tonight?", the LLM needs search results data, not website content.

SerpApi's JSON responses are just as "LLM-ready" as Firecrawl's markdown, arguably more so because they're pre-structured. When you integrate SerpApi with OpenAI's Assistant API, your LLM can query real-time search data and receive structured answers that Firecrawl simply cannot provide because it doesn't access Google Maps or structured search results.

SerpApi as LLM's web search API

See how SerpApi can be integrated with various LLMs.

SerpApi vs Firecrawl: Feature-by-Feature Comparison

Here's how SerpApi vs Firecrawl stack up across the dimensions that matter for search intelligence:

Feature	SerpApi	Firecrawl
Primary Function	Scrapes search engine results pages (Google, Bing, etc.)	Scrapes individual website content
Data Type	Search rankings, ads, prices, local results, SERP features	Website text, markdown, HTML content
Search Engine Access	17+ platforms, 100+ API endpoints	Basic search endpoint (not true SERP scraping)
Ranking Data	Full organic and paid ranking positions	Not applicable
Shopping Data	Google Shopping prices, sellers, reviews, positions	Must scrape individual retailer sites
Local Business Data	Google Maps with reviews, hours, photos, Q&A	Must scrape business websites individually
SERP Features	Featured snippets, People Also Ask, Knowledge Graph, 40+ feature types	Not applicable
LLM Integration	Structured JSON for AI consumption	Markdown optimized for LLMs
Response Time	~0.73-1.75 seconds with Ludicrous Speed	Varies by page complexity
Pricing Model	Per search query (starts at $25/month for 1,000 searches)	Per page crawled (starts at $16/month for 3,000 credits)
Use Case Focus	Competitive intelligence, SEO, market research	Content extraction, documentation

Understanding the Pricing Models

The pricing structures reflect the fundamental difference in what each tool does. SerpApi charges per search query because each query hits a search engine and extracts structured results. A single query like "best CRM software" can return many data points (10 organic results, ads, related questions, etc.).

Firecrawl charges per page crawled because it's extracting content from individual URLs. A single-page scrape returns the markdown content of that one page.

They're not comparable on price because they're not comparable on function. If you need SERP data, you cannot use Firecrawl regardless of its pricing.

On top of that, SerpApi gets cheaper as you scale.

When to Use Each Tool

Choose Firecrawl when:

Your data lives on specific websites, and you know the URLs
You need full-text content for LLM consumption
You're building content aggregation or documentation scrapers
Your use case is "scrape these websites"

Choose SerpApi when:

Your data lives in search engine results, and you need to query by keyword
You need search visibility metrics, rankings, or SERP features
You're building SEO tools, price monitoring, or competitive intelligence
Your use case is "what does Google show when I search for X"

Conclusion: Choose Based on Your Data Source

The decision between SerpApi vs Firecrawl isn't about which is "better." It's about which data source your application requires.

The web has two distinct data layers: the content that lives on websites, and the metadata that search engines display about that content. Firecrawl accesses the first. SerpApi accesses the second. For business intelligence, competitive analysis, and understanding market dynamics, the second layer is what matters.

If you're building SEO tools, price monitoring, competitive intelligence, or local business analysis, you need search engine data. That's where SerpApi operates, and that's why it's the tool of choice for developers who need to understand not just what exists on the web, but what search engines choose to show their users.

Ready to build with real search intelligence? Start your free trial and access the data that drives business decisions.

Next Steps

After understanding the fundamental differences between SerpApi and Firecrawl, here are practical ways to get started with real-world search data:

Explore Search Data Types : Try SerpApi's interactive playground to see structured results from Google Search, Google Shopping, Google Maps, and many other platforms. Compare how different search engines structure their results.

Build Rank Tracking : Follow our guide to create a SERP tracking API and monitor keyword positions across locations.

Extract Competitive Intelligence : Learn to scrape Google Ads data to analyze competitor ad copy and bidding strategies. Track Google Shopping results for real-time price intelligence.

Integrate with AI Applications : Connect SerpApi to OpenAI's Assistant API for real-time search capabilities, or build an AI-powered SEO research agent that analyzes search trends automatically.

Access Local Business Data : Master Google Maps scraping for local SEO analysis and competitive research. Build location-based intelligence tools that track reviews, ratings, and local pack rankings.

Also, check various SerpApi's use cases.

SerpApi vs. Exa.ai: The Difference Between "Real-Time Truth" and "Neural Search"

Hilman Ramadhan — Sun, 26 Jul 2026 11:14:06 +0000

If you're building an AI agent or a tool that needs search data, you've probably seen Exa.ai pop up in your research. The marketing positioning is compelling: "AI-native semantic search." But here's the question that matters: does your application need "an" answer or "the" answer?

Exa describes itself as "the first meaning-based web search API powered by embeddings." What this actually means is that Exa searches its own proprietary index of the web using neural embeddings. It's like having a well-organized library catalog. You get results quickly because you're just querying a database.

Exa.ai vs SerpApi

SerpApi scrapes Google, Bing, and other search engines in real-time. It's not searching a catalog. It's reading the live dashboard of what search engines are displaying to users right now - like checking an instrument panel that shows current readings, not historical records. That's why SerpApi delivers real-time search intelligence, not cached snapshots.

This isn't a subtle difference. If you're building a RAG chatbot that needs to answer generic questions like "explain quantum computing," Exa might work fine. But if you're building anything that requires market intelligence, SEO rankings, price monitoring, ad verification, or local search results, you need real-time data that reflects the actual state of search engines. That's what SerpApi delivers.

This article breaks down the technical and business differences between these two approaches so you can make an informed decision.

Also, read the_ _best Web Search API options in 2026.

Data Freshness and "The Truth"

The core architectural difference between Exa and SerpApi is what they're actually querying. Exa uses a proprietary index of the web, trained on embeddings for semantic search. This means Exa crawls the web periodically, builds an index, and then serves queries against that index. The problem with this approach is staleness.

Exa landing page

Here's a concrete example. Let's say a user asks your AI agent, "What's the price of the iPhone 17 right now?" If you query Exa, you might get a blog post that mentions the iPhone 17's price, but that post could be from three days ago. The price might have changed. The product might be out of stock. You won't know because Exa is showing you what was true when it last crawled that page.

If you query SerpApi's Google Shopping API, you get the actual Google Shopping results. You're seeing the same data that appears when a user searches Google Shopping in their browser. That's real-time truth.

Sample iPhone 17 price search on Exa and SerpApi

This matters even more for SEO and local search. If you're building a rank tracking tool, your customers want to know where their site ranks today, not where it ranked when Exa last updated its index. SerpApi captures search results exactly as they appear in real-time, including Google Maps, Local Packs, Knowledge Graphs, and Shopping Carousels. Exa doesn't support any of these structured result types because it's searching its own index, not Google's live results.

Let's look at what data types each service actually supports:

Exa's Coverage:

Text content from web pages
Semantic search across its index
Basic filtering (date, domain, category)

SerpApi's Coverage:

Organic search results from Google, Bing, Yahoo, Baidu, Yandex, DuckDuckGo
Google Maps and Local Pack results
Shopping results (Google Shopping, Amazon, eBay, Walmart, Home Depot)
Image and video results
Knowledge Graphs and answer boxes
News results
Google Flights and Hotels
Google Trends

If your use case requires anything beyond basic text search, Exa won't work. You can't get local business data, you can't track shopping prices, you can't monitor ad placements. Those data types don't exist in Exa's index.

Performance Wars: "Neural Speed" vs. Real-Time Scraping

Critics point out that scraping Google takes longer than querying a database. That's technically true. Exa can be faster because you're just doing a database lookup.

But here's what that criticism misses: SerpApi has closed the performance gap. SerpApi's measured 30-day average response time is 1.09 seconds. For the majority of requests, you're looking at 1-2 second response times.

For speed-sensitive applications, SerpApi offers Ludicrous Speed mode (using 2x server resources) and Ludicrous Speed Max mode (using 4x server resources) that can deliver sub-second response times while maintaining the same real-time data quality. This means you can get speed comparable to Exa's database lookups without sacrificing data freshness.

Let's put this in perspective. If you're building a chatbot that responds to user queries, a 1-2 second difference in response time is usually acceptable. Users expect a slight delay when they ask a question that requires a web search. What they don't expect is outdated or incomplete information.

The real question is: would you rather have a faster response with potentially stale data, or a response with real-time, accurate data? For most enterprise use cases, the answer is obvious.

SerpApi also offers performance optimization options for speed-sensitive applications. The Google Light Fast API can achieve faster response times depending on your specific needs and the complexity of the search query.

Here's the performance breakdown:

SerpApi:

Average response time: ~1.09s (30-day measured average)
Ludicrous Speed modes available for sub-second performance
Success rate: 99.95% SLA

Exa:

Fast mode: 178ms P50, 252 P90

The other performance metric that matters is reliability. SerpApi maintains a 99.95% success rate across all requests. When you make a request, you get a result. That reliability matters when you're building production systems.

Feature-by-Feature Comparison

Here's a direct comparison table to help you evaluate the two services:

Feature	SerpApi	Exa
Data Source	Real-time scraping of 50+ search engines and platforms	Proprietary internal index
Search Engine Coverage	✅ Google, Bing, Yahoo, Amazon, eBay, Walmart, Yandex, Baidu, DuckDuckGo, Naver, and 40+ more	❌ Single proprietary index only
Freshness	Real-time (live mirror of search engines)	Updated every minute (news); incremental for other content
Supported Data Types	Organic results, Maps, Local Pack, Shopping, Images, Videos, Flights, Hotels, Knowledge Graphs, News, Ads	Text content from web pages, semantic search
Structured SERP Features	✅ Knowledge Graphs, Local Pack, Shopping Carousels, Answer Boxes, Featured Snippets, People Also Ask, Related Searches	❌ Limited to text content
E-commerce Platform Support	✅ Amazon, eBay, Walmart, Home Depot APIs	❌ None
Ad Data	✅ Paid search ads included in all results	❌ Not available
Local Business Data	✅ Google Maps, reviews, photos, hours, contact info	❌ Not available
Reliability	99.95% SLA (actual uptime: 99.967%)	Not publicly disclosed
Location Targeting	✅ Precise geo-targeting with 1000+ locations worldwide	✅ Worldwide coverage (less granular)
Device-Specific Results	✅ Mobile, tablet, desktop variations	❌ Not supported
CAPTCHA Handling	✅ Automatic CAPTCHA solving included	N/A (no scraping)
Browser Rendering	✅ Full browser simulation for JavaScript-heavy pages	N/A (index-based)
Historical Data	✅ Search Archive API available	❌ Not available
Caching	✅ Built-in (free repeated queries)	N/A
API Architecture	✅ JSON output, REST API	✅ JSON output, REST API
Client Libraries/SDKs	✅ 10+ languages (Python, Node.js, Ruby, PHP, Java, Go, Rust, .NET, Swift, C++)	✅ Python, JavaScript, curl
No-Code Integration	✅ Google Sheets add-on, Zapier, Make.com, n8n, Bubble	❌ API-only (requires coding)
Interactive Playground	✅ Test queries in browser before coding	✅ Available
Documentation	✅ Comprehensive docs and SDKs	✅ Comprehensive docs and SDKs
Enterprise Options	✅ Custom SLAs available (99.97% for enterprise)	✅ Custom SLAs available
Global Access	✅ Multi-region proxy networks	✅ Worldwide index coverage
Legal Protection	✅ US Legal Shield with up to $2M indemnification (Production+ plans)	❌ None (customers must indemnify Exa)
Pricing (1k searches)	Starts at $75/month for 5,000 searches. Getting cheaper as you you scale	$5 (Basic Search 1-25 results), $15 (Deep Search)
Use Cases	Data for AI Agent, SEO tracking, price monitoring, ad verification, local search, market intelligence, competitive analysis	Generic RAG, content discovery, semantic search, concept exploration

The pricing comparison requires context. Exa is cheaper on a per-search basis for basic semantic queries ($5 per 1,000 searches vs. SerpApi's $10-25 per 1,000 at standard tiers). However, SerpApi gets cheaper as you scale. When you factor in the value of real-time data, comprehensive coverage of SERP features, and legal protection, SerpApi's pricing makes sense for production use cases that require search engine accuracy.

Which API Is Right for You?

Choose Exa if you're building a system that benefits from semantic understanding and concept exploration, where speed is critical and data freshness within minutes or hours is acceptable. Exa works well for RAG applications and content discovery tools.

Choose SerpApi if you're building a business that needs data reflecting the actual state of search engines right now. If you need web search data for AI tools, SEO rankings, product prices, ad placements, local results, or any structured SERP features beyond basic text, SerpApi is the only option. If you're an enterprise that needs legal protection for data collection activities, SerpApi provides the US Legal Shield feature.

The fundamental question is this: are you building a system that needs to understand what the web says about a topic (Exa's strength), or are you building a system that needs to understand what search engines are showing users right now (SerpApi's strength)? For most production applications involving market intelligence or user-facing search data, you need the latter.

Don't settle for a "version" of the web when you need real-time accuracy. SerpApi has provided developers, researchers, and businesses with access to real-time search data for over eight years.

Next Steps

Once you've chosen your search API approach, here are the logical next steps based on your use case:

For SEO and Rank Tracking Projects:

Start building your rank tracking system using SerpApi's SERP Tracking API. This guide walks through creating a white-label rank tracker application that monitors search rankings in real-time.

For E-commerce and Price Monitoring:

Implement automated price tracking with the Amazon Price Tracking tutorial. Learn how to monitor product prices over time and detect market trends across multiple platforms.

For Multi-Platform Search Applications:

If you're migrating from another search API or expanding your data sources, review the Bing Search API Replacement guide to understand how SerpApi provides comprehensive coverage across search engines.

For Local Business Intelligence:

Build location-based applications using the Google Maps Data Scraping tutorial. This covers extracting business listings, reviews, and local pack data for market analysis.

For API Optimization and Cost Management:

Understand how to optimize your search credit usage with the Credit Usage Calculation guide. Learn which parameters affect credit consumption and how to structure queries efficiently.

For Technical Implementation:

Master the full range of search customization options in the Google Search Parameters documentation. This comprehensive reference covers all available parameters for fine-tuning your search queries.

SerpApi vs. Brave Search API: Why "Independent" Data Isn't Enough for Business Intelligence

Hilman Ramadhan — Sat, 25 Jul 2026 11:09:10 +0000

When comparing SerpApi vs Brave Search API, the fundamental question isn't about data quality in isolation; it's about data relevance. The Brave Search API queries a private, independent index of the web containing approximately 20 billion pages. Compare that to Google's index, which spans hundreds of billions of pages. Brave bills itself as privacy-first and independent from Big Tech. It's what you might call a "Sanitized Web", curated, lean, and intentionally isolated from the mainstream search ecosystem.

Brave Search API vs SerpApi

SerpApi takes a different approach. It scrapes the live results from Google, Bing, YouTube, and yes, even Brave if you need it. It's the "Real Web", the messy, commercial, algorithmically-ranked reality that your customers actually see when they search.

For generic retrieval-augmented generation (RAG) tasks like "What is the capital of France?" or "Explain photosynthesis," Brave works fine. The answer doesn't change based on which index you query. But for Business Intelligence questions like "Who is bidding on 'CRM software' in London?" or "What's the average price for wireless earbuds on Google Shopping right now?" Brave becomes statistically irrelevant.

The core issue isn't that Brave is bad. It's that Brave answers a different question. If your business depends on understanding what consumers actually see, where they click, what prices they compare, which local businesses appear in the map pack, then you're not just choosing between two APIs. You're choosing between building on top of the market or building in parallel to it.

The 1% vs. 90% Problem: Understanding Brave Search API Limitations

When evaluating Google Search API vs Brave, the market share statistics tell a sobering story. Google controls approximately 90% of the global search market. Bing sits at around 4%. Brave Search, despite its privacy credentials and growing user base, represents less than 1% of search traffic.

This creates what I call the "Vanity Metric" danger, one of the most critical Brave Search API limitations for commercial applications. Imagine you're building SEO rank tracking APIs for marketing agencies. Your clients want to know where their e-commerce site ranks for "running shoes" in New York. If you build this tool using the Brave Search API, you're reporting on a user base that effectively doesn't exist for most businesses.

Your dashboard shows their client is ranking #3 on Brave. That sounds great until you realize that Brave's <1% market share means the client is visible to a tiny fraction of potential customers. Meanwhile, on Google, where 90% of searches happen and where actual revenue gets generated, that same site might be on page 10. Your tool just gave them false confidence based on irrelevant data.

Example of different search results between Google Search and Brave Search

The problem compounds when you consider the algorithm gap. Brave's ranking algorithm is completely independent from Google's. It doesn't use PageRank, doesn't weight E-E-A-T signals the same way, and doesn't apply the same spam filters or helpful content system updates that Google deploys. The two search engines are fundamentally solving different optimization problems.

This means optimizing for Brave does not help you rank on Google. If you're an SEO tool provider giving recommendations based on Brave data, you're telling your users how to rank on a platform their customers don't use. There's no feedback loop between Brave performance and business outcomes because the traffic isn't there.

Only SerpApi gives you access to the ranking signals, ad positions, and SERP features from the platforms where revenue actually happens. When your customer asks "Why did our traffic drop 40% last week?" you need to see what Google changed in their SERP, not what Brave might hypothetically show to its tiny user base.

The use case for Brave data in SEO tooling is essentially academic. It's interesting for research on algorithmic diversity or privacy-preserving search. But if you're building a business that needs to predict, track, or optimize for search visibility that translates to revenue, you need Google data;. That's not an opinion, it's arithmetic.

The "Rich Result" Gap: Where Brave Search API Limitations Become Critical

Brave is primarily a text-link search engine. You query it, you get a list of URLs with titles and descriptions. For many use cases, that's sufficient. But the modern Google SERP is nothing like a list of ten blue links.

Google's "rich results" now dominate commercial queries. These include local packs, shopping carousels, knowledge panels, video results, featured snippets, and "People Also Ask" boxes. These elements aren't decorative; they're where clicks happen. Featured snippets can capture significant traffic on informational queries, often pulling clicks away from traditional organic results.

Brave has none of this. Here's what SerpApi captures that Brave misses:

Local SEO Data API Capabilities

Local Data: Brave has no equivalent to Google's Local Pack. When you search "coffee shop near me" on Google, you get a map, business hours, real-time busy times, photos, and review scores. SerpApi connects to Google Maps data, pulling these structured signals in real-time. If you're building a local SEO data API solution, a competitive analysis tool for restaurants, or a location intelligence platform, Brave literally cannot provide the data you need. It doesn't have relationships with local business databases or review platforms at the scale Google has built through Google Business Profile.

Shopping Data: Brave has no Shopping Graph. When you search "Sony WH-1000XM5" on Google, you see prices from a dozen retailers, shipping estimates, availability, and product ratings aggregated across the web. SerpApi scrapes Google Shopping to give you real-time price comparisons, seller reputations, and stock levels. If you're building a price monitoring tool, a dropshipping product research platform, or an AI shopping assistant, Brave gives you a link to Sony's website. SerpApi provides you with the competitive market landscape.

Visual Search: Brave's image search is generic web crawl data. SerpApi scrapes Google Lens and reverse image search, which use sophisticated computer vision models to identify products, landmarks, plants, and more. The difference is the layer of machine learning that Google applies to visual data, something Brave's index simply doesn't have at a comparable scale or accuracy.

Real-World Use Case Comparison

Here's a concrete example. You're building an AI travel agent. A user asks, "Find me a hotel in Barcelona near Park Güell under $150 per night."

Using Brave Search API, your agent searches "hotels near Park Güell Barcelona" and gets back a list of hotel websites and booking aggregators. It might scrape a few of those sites to find prices, assuming they're not blocked. You get generic results with no structured data.

Using SerpApi's Google Hotels integration, your agent gets back structured JSON with exact prices per night, availability for specific dates, review scores, photos, amenities, distance to Park Güell calculated by Google Maps, and direct booking links. You can filter by price programmatically before even showing results to the user.

The first implementation requires you to build your own scraping infrastructure for dozens of hotel sites. The second gives you production-ready data in one API call. This isn't a minor difference; it's the difference between a prototype and a product customers will pay for.

Performance & "Ludicrous Speed"

Let's acknowledge the elephant in the room: Brave is fast. Query response times are typically under 500 milliseconds because you're hitting a purpose-built API on Brave's infrastructure. There's no scraping, no proxy rotation, no parsing complexity.

SerpApi is scraping live web pages, which adds latency. That's a trade-off.

But here's the counter: SerpApi's Ludicrous Speed Max brings Google scraping down to approximately 1-2 seconds. For most business intelligence use cases, that's fast enough. If you're running hourly rank checks, daily price monitoring, or enriching a CRM with company data pulled from search, 1-2 seconds is completely acceptable.

Ask yourself this: Is saving 800 milliseconds worth missing 90% of the market? Is it worth losing access to local data, shopping data, and rich results that drive actual business decisions?

For real-time, user-facing search where latency is critical, consider using Brave as a fallback or for generic queries. But for the business intelligence layer, the data that informs product decisions, marketing spend, and competitive strategy, the marginal latency of SerpApi is worth the accuracy and completeness.

Speed is a feature. Relevance is a requirement.

SerpApi vs Brave Search API: Feature-by-Feature Comparison

Here's how SerpApi vs Brave Search API stack up across the dimensions that matter for business intelligence:

Feature	SerpApi	Brave Search API
Primary Data Source	Live scraping of Google, Bing, Yahoo, DuckDuckGo, Yandex, Baidu, YouTube, and 50+ sources	Brave's independent index (~20B pages)
Market Coverage	~90% (Google's market share)	<1% (Brave's market share)
Shopping Data	Full Google Shopping integration with prices, sellers, shipping, reviews	No shopping-specific data
Local Business Data	Google Maps API with reviews, photos, hours, busy times, Q&A	No local pack or structured business data
Rich Results	Featured snippets, People Also Ask, Knowledge Graph, video carousels	Basic organic results only
Legal Protection	$2M indemnification via US Legal Shield	Standard API terms (no scraping protection)
Average Response Time	Depend on the API and the speed mode	<500ms
Pricing	Starts at $75/month for 5,000 searches. Getting cheaper as you you scale	$5 per 1,000 queries (~$0.005 per query)
Supported Search Engines	50+ including Google, Bing, Yahoo, DuckDuckGo, Brave, Yandex, Baidu	Brave only
Image Search	Google Lens, reverse image search with ML identification	Generic image index

Understanding SerpApi Pricing

The pricing difference is real. Brave is more affordable per query at face value. But when evaluating SerpApi pricing, consider the value proposition. If those Brave queries return data that doesn't reflect your customers' reality, you're not saving money; you're buying irrelevant data at a discount. SerpApi's pricing structure reflects the complexity of delivering real-time, structured data from the search engines where actual business happens.

Conclusion: Do You Want "Clean" Data or "Right" Data?

Brave is a great backup search engine. It's a solid training dataset for generic knowledge retrieval. If you're building a RAG system for a chatbot that answers encyclopedia-style questions, Brave will work fine.

But Brave is an island. Its index is independent, its algorithm is independent, and its user base is a rounding error in the global search market. If your business depends on understanding what consumers see when they search, which is Google 90% of the time, you cannot afford to look at a different index.

The data might be clean, but it's not right. It's not representative. It doesn't show you where your customers' ads appear, where their competitors rank, what prices are displayed, or which local businesses get the map pack visibility.

When choosing between Google Search API vs Brave , or more accurately, between SerpApi (which provides Google data) and Brave's independent index, the choice becomes clear. SerpApi is the universal adapter for the actual digital economy. It connects to Google, where the revenue happens, Bing, where enterprise users search, YouTube, where product reviews live, Amazon, where e-commerce pricing is set, and yes, even Brave if you need it for comparison. You're not locked into one provider's view of the web; you get the web as it actually exists across platforms.

If you're building SEO rank tracking APIs , price monitoring , local SEO data API solutions, ad intelligence, or competitive analysis tools, your users don't care about Brave's independent index. They care about Google's SERP because that's where their traffic comes from. Give them data from the source that matters.

See the market as it really is. Get 100% accurate Google data with SerpApi. Start your free trial and query the real web, not a sanitized approximation.

Also read best Web Search API options in 2026.

Next Steps

After understanding the fundamental differences between SerpApi and Brave Search API, here are practical ways to get started with real-world search data:

Start Building with SerpApi : Try SerpApi's interactive playground to test queries and see structured results immediately. Explore our integration libraries for Python, Node.js, PHP, Ruby, Java, and other languages to get production-ready code snippets.

Track Your Rankings : Build a rank tracking application to monitor keyword positions across locations and devices. Learn how to calculate search credit usage to optimize your monitoring strategy.

Extract Competitive Intelligence : Discover how to scrape Google Ads data from competitors and analyze shopping results to understand pricing strategies in your market.

Build Location-Based Tools : Learn to scrape Google Maps data and reviews for local business intelligence, or create a hotel price tracker with our Google Hotels API.

Integrate with AI Workflows : Connect SerpApi to OpenAI's Assistant API for real-time web knowledge, or build an AI-powered SEO research agent that analyzes search trends automatically.

SerpApi is the Web Search API you need for your applications and AI agents.

SerpApi Alternatives: Best Web Search APIs in 2026

Hilman Ramadhan — Fri, 24 Jul 2026 09:39:07 +0000

If you’re building products that rely on web search — AI apps, research tools, lead collectors, price trackers, or internal dashboards — you’ve probably heard of SerpApi. It’s a popular choice because it turns messy search engine pages into clean, structured data with zero scraping pain.

That said, SerpApi isn’t the only option. You may have heard some of the Search API alternatives. In this article, we’ll walk through the most popular SerpApi alternatives as a Web Search API:

Brave Search API
Exa
Tavily
Firecrawl
Serperdev and more

We’ll break down what each tool is good at, who it’s best for, and when you might choose it over SerpApi.

SerpApi alternatives as a Web Search API

Why do many companies choose SerpApi?

Before jumping into alternatives, here’s why developers typically use SerpApi:

Complete Search Engine APIs: We cover more than 100 APIs that give you programmatic access to Google, Bing, DuckDuckGo, etc.
Real-time results: We always perform live scraping for you; it's not coming from a database.
Structured JSON responses (no HTML parsing). No more DIY that forces you to spend time and resources on keeping up to date with the latest layout. Use a simple API, and you are done.
Enterprise-ready: We're SOC2 Type II certified and can handle a high volume of searches for your team.
We've been in business since 2017. So, we faced many issues that our competitor hasn't. We're focusing on scraping search engine results, not just any websites. That keeps us focused on our vision and our daily work.
Location, language, device, and other parameters to narrow your search to what you need.
Use cases: AI agents, SEO tools, market research, price monitoring, and lead collection.

If all of that sounds good to you. You can register for free at serpapi.com

Best Web Search API Options

Now, let's see some options for web search API:

1. SerpApi

SerpApi provides public search results data from various search engines in a structured JSON format. We currently cover more than 100+ APIs.

We don't manipulate the data in any way; we leave it to the developers to decide what to do with it, including providing responses to AI tools for various use cases. No hidden algorithm.

SerpApi - web search API

Pros

Unedited real-time results
Support 100+ APIs, you can pick depending on your application needs
Simple API through a single HTTP GET request
Flexible use cases

Cons

Depending on the volume, SerpApi is not the cheapest option, as it prioritizes result quality.

2. Brave Search API

Brave Search API gives you access to Brave’s own search index, not Google or Bing results.

Pros

Uses Brave’s independent search index
Want an alternative to Google/Bing dependency

Cons

Limited to only Brave's own algorithm and database
Not a drop-in replacement for Google SERP and other engines scraping
Not ideal for product or business research
Less SEO-specific data (ads, rich snippets, local packs, etc.)

Brave Search API alternatives

When to choose Brave over SerpApi

👉 When your apps need generic knowledge from search results and want to be independent from Google, Bing, or other search engines.

3. Exa

Instead of returning raw SERP-like results, Exa focuses on semantic search or "natural language search query".

Pros

Uses Exa's own search index. It can be faster as the source is coming from a database

Cons

Not designed to perform live scraping on search engines
Limited to their own algorithm and database
Not ideal for product or business research
Less useful for SEO monitoring, rank tracking, or ad data

Exa AI Search API Alternatives

When to choose Exa over SerpApi

👉 When you’re building apps that don't really need real-time or raw data from a search engine.

4. Tavily

Tavily takes a different approach. It aggregates content from multiple web sources and processes it on its side before returning the response to the customer.

Pros

Aggregated search results

Cons

Returned results are based on their algorithm
Summarized search responses might not be what people want in many cases
Not ideal for product research or SEO tooling that needs real-time data.

Tavily alternatives

When to choose Tavily over SerpApi

👉 When your app doesn't need the exact live data results from search engines, and you're fine using Tavily's algorithm for the returned results.

5. Firecrawl

Firecrawl is a website content scraper. You point it at a URL, techcrunch.com/article/some-news and it returns clean, LLM-ready markdown of that article.

SerpApi is a search engine scraper. You query it with "best CRM software," and it returns what Google (and other top search engines) shows for that query: the top 10 organic results, the paid ads, the featured snippet, the "People Also Ask" box, and the local map pack.

Pros:

Return the full content of a link

Cons:

You have to supply the individual link/URL
You can't scrape specific search engine results

6. Serperdev

Serperdev is closer to SerpApi in how they return raw search engine results.

Please read our blog post on SerpApi vs Serper:

Alternatives to SerpApi 2026 (Serper, SearchApi, and ValueSERP)

When to choose Serper over SerpApi

👉 When you want something closer to traditional SERP scraping with simpler needs or lighter usage.

Final Thoughts

Every web search API has its place. Tools like Brave, Exa, Tavily, or others shine for specific use cases.

If your product depends on reliable, production-grade SERP data at scale across search, shopping, maps, reviews, and more, long-term stability matters. As you move from prototypes to real users, mature infrastructure, consistent parsers, strong uptime, and predictable data formats become critical.

This is where SerpApi stands out. It’s more than a web search API. It’s a battle-tested web data platform teams trust to power AI features, SEO tools, lead generation, price monitoring, and product research.

If you’re evaluating options, the best move is to try a few APIs side by side. But when reliability, coverage, and developer experience start to matter more than experiments, you’ll likely want something built for the long haul.

How to Scrape Google Flights Deals with a simple API

Hilman Ramadhan — Tue, 14 Jul 2026 06:34:07 +0000

Google Flights Deals helps travelers discover discounted flights without a specific destination. Instead of searching one route at a time, you can start with a departure airport or city and explore discounted destinations.

This data can be useful for building flight-deal trackers, travel discovery tools, price-comparison dashboards, or destination-recommendation apps. In this tutorial, we'll use SerpApi's Google Flights Deals API to retrieve flight deals as structured JSON with cURL, Python, and JavaScript.

Scrape Google Flights Deals website

How to use the Google Flights Deals API

A basic request uses these parameters:

Parameter	Requirement	Description
`engine`	Required	Set to `google_flights_deals`
`api_key`	Required	Your private SerpApi API key
`departure_id`	Optional	Departure airport code or location ID
`currency`	Optional	Currency used for returned prices

An airport ID is normally an uppercase three-letter IATA code such as:

CDG for Paris Charles de Gaulle Airport
AUS for Austin-Bergstrom International Airport
MAD for Adolfo Suárez Madrid-Barajas Airport

You can also provide a Google location ID beginning with /m/ or /g/. Multiple departure locations can be separated with commas.

Get your API key

Create a SerpApi account and retrieve your private key from the API key page.

Replace the following placeholder in each example:

YOUR_SERPAPI_API_KEY

Never expose your API key in public source code or commit it to a public repository.

The complete parameter reference is available in the official Google Flights Deals API documentation.

cURL implementation

Let's retrieve flight deals departing from Paris Charles de Gaulle Airport:

curl --get https://serpapi.com/search \
 -d engine="google_flights_deals" \
 -d departure_id="CDG" \
 -d currency="USD" \
 -d api_key="YOUR_SERPAPI_API_KEY"

The parameters in this request are:

engine=google_flights_deals selects the Google Flights Deals engine.
departure_id=CDG searches for deals departing from Paris Charles de Gaulle Airport.
currency=USD returns prices in US dollars.

The API returns JSON by default.

Python implementation

Create a new Python file:

main.py

Install the requests package:

pip install requests

Add the following code:

import requests


SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "engine": "google_flights_deals",
    "departure_id": "CDG",
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

response = requests.get(
    "https://serpapi.com/search",
    params=params,
    timeout=30,
)
response.raise_for_status()

data = response.json()

if "error" in data:
    raise RuntimeError(data["error"])

departure = data.get("departure_informations", {})

departure_name = (
    departure.get("airport_name")
    or departure.get("city")
    or "Unknown departure"
)

print(f"Flight deals from: {departure_name}")
print()

for deal in data.get("deals", []):
    duration = deal.get("flight_duration")

    if duration is not None:
        hours, minutes = divmod(duration, 60)
        formatted_duration = f"{hours}h {minutes}m"
    else:
        formatted_duration = "Not available"

    print("Destination:", deal.get("name", "Unknown"))
    print("Country:", deal.get("country", "Unknown"))
    print("Price:", deal.get("price", "Not available"))
    print(
        "Average price:",
        deal.get("average_price", "Not available"),
    )
    print(
        "Discount:",
        deal.get("discount_percentage", "Not available"),
    )
    print("Outbound date:", deal.get("outbound_date", "Not available"))
    print("Return date:", deal.get("return_date", "One-way flight"))
    print(
        "Route:",
        f"{deal.get('departure_airport_code', '?')} → "
        f"{deal.get('arrival_airport_code', '?')}",
    )
    print("Duration:", formatted_duration)
    print("Stops:", deal.get("stops", "Not available"))
    print("Airline:", deal.get("airline", "Not available"))
    print("Google Flights:", deal.get("flight_link", "Not available"))
    print("-" * 80)

The code prints selected fields from every item in the deals array. It also converts the documented flight_duration value from minutes into hours and minutes.

Here is the result

Flights Deals API scraper

Fields are accessed with .get() because some values are not guaranteed to appear in every deal.

Export the flight deals to CSV

You can also save the results to a CSV file for further analysis:

import csv


rows = []

for deal in data.get("deals", []):
    rows.append({
        "destination": deal.get("name", ""),
        "country": deal.get("country", ""),
        "destination_id": deal.get("destination_id", ""),
        "price": deal.get("price", ""),
        "average_price": deal.get("average_price", ""),
        "discount_percentage": deal.get(
            "discount_percentage",
            "",
        ),
        "outbound_date": deal.get("outbound_date", ""),
        "return_date": deal.get("return_date", ""),
        "departure_airport": deal.get(
            "departure_airport_code",
            "",
        ),
        "arrival_airport": deal.get(
            "arrival_airport_code",
            "",
        ),
        "flight_duration_minutes": deal.get(
            "flight_duration",
            "",
        ),
        "stops": deal.get("stops", ""),
        "airline": deal.get("airline", ""),
        "airline_code": deal.get("airline_code", ""),
        "google_flights_link": deal.get("flight_link", ""),
        "serpapi_flight_link": deal.get(
            "serpapi_flight_link",
            "",
        ),
    })

fieldnames = [
    "destination",
    "country",
    "destination_id",
    "price",
    "average_price",
    "discount_percentage",
    "outbound_date",
    "return_date",
    "departure_airport",
    "arrival_airport",
    "flight_duration_minutes",
    "stops",
    "airline",
    "airline_code",
    "google_flights_link",
    "serpapi_flight_link",
]

with open(
    "google_flights_deals.csv",
    "w",
    newline="",
    encoding="utf-8",
) as csv_file:
    writer = csv.DictWriter(
        csv_file,
        fieldnames=fieldnames,
    )
    writer.writeheader()
    writer.writerows(rows)

print(
    f"Saved {len(rows)} deals to "
    "google_flights_deals.csv"
)

This creates a file named google_flights_deals.csv with one row for each deal.

Exported result in CSV file

JavaScript implementation

SerpApi provides an official JavaScript package.

Install it with npm:

npm install serpapi

Create an index.js file:

const { getJson } = require("serpapi");

const SERPAPI_API_KEY = "YOUR_API_KEY";

getJson(
  {
    engine: "google_flights_deals",
    departure_id: "CGK",
    currency: "USD",
    api_key: SERPAPI_API_KEY,
  },
  (json) => {
    if (json.error) {
      console.error(json.error);
      return;
    }

    const departure = json.departure_informations || {};
    const deals = json.deals || [];

    console.log(
      `Flight deals from: ${
        departure.airport_name ||
        departure.city ||
        "Unknown departure"
      }\n`
    );

    deals.forEach((deal) => {
      console.log({
        destination: deal.name || "Unknown",
        country: deal.country || "Unknown",
        price: deal.price ?? "Not available",
        averagePrice:
          deal.average_price ?? "Not available",
        discountPercentage:
          deal.discount_percentage ?? "Not available",
        outboundDate:
          deal.outbound_date || "Not available",
        returnDate:
          deal.return_date || "One-way flight",
        route:
          `${deal.departure_airport_code || "?"} → ` +
          `${deal.arrival_airport_code || "?"}`,
        duration: deal.flight_duration ?? "Not available",
        stops: deal.stops ?? "Not available",
        airline: deal.airline || "Not available",
        flightLink: deal.flight_link || "Not available",
      });
    });
  }
);

Run the script:

node index.js

The official SerpApi JavaScript package exposes the getJson function and supports both callbacks and promises.

Customize Google Flights Deals results

The API provides several ways to customize the returned deals. Here are three useful examples.

Search using exact or flexible dates

For an exact round trip, provide one outbound_date and one return_date in YYYY-MM-DD format:

params = {
    "engine": "google_flights_deals",
    "departure_id": "FCO",
    "outbound_date": "2026-08-10",
    "return_date": "2026-08-17",
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

For flexible dates, use two comma-separated dates as the departure window:

params = {
    "engine": "google_flights_deals",
    "departure_id": "BCN",
    "outbound_date": "2026-09-01,2026-09-30",
    "travel_duration": 3,
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

Possible travel_duration values are:

Value	Duration
`1`	One week
`2`	Weekend
`3`	Two weeks

You can alternatively use trip_length for a custom duration:

params["trip_length"] = "5,10"

This requests trips lasting between 5 and 10 days.

travel_duration, trip_length, and return_date cannot be used together. For a round trip with one exact outbound date, one exact return date is required.

Filter by stops and maximum price

Use stops to control how many connections are allowed:

Value	Meaning
`0`	Any number of stops
`1`	Nonstop only
`2`	One stop or fewer
`3`	Two stops or fewer

You can combine it with max_price:

params = {
    "engine": "google_flights_deals",
    "departure_id": "CDG",
    "stops": 1,
    "max_price": 500,
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

This searches for nonstop deals costing no more than 500 in the selected currency.

Include or exclude airlines

Use include_airlines to return deals from selected airlines:

params = {
    "engine": "google_flights_deals",
    "departure_id": "/m/0947l",
    "include_airlines": "W4,FR",
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

The example includes:

W4 — Wizz Air
FR — Ryanair

Multiple airline codes are separated with commas.

You can use exclude_airlines instead when you want to remove specific airlines from the results:

params["exclude_airlines"] = "UA"

The include_airlines and exclude_airlines parameters cannot be used together.

The API also accepts the following alliance values:

STAR_ALLIANCE
SKYTEAM
ONEWORLD

Searching for one-way flight deals

Round trip is the default flight type. Set type to 2 to search for one-way deals:

params = {
    "engine": "google_flights_deals",
    "departure_id": "MAD",
    "type": 2,
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

The supported values are:

Value	Flight type
`1`	Round trip
`2`	One way

For a one-way search with a specific date, provide only outbound_date:

params["outbound_date"] = "2026-09-15"

Do not use return_date, travel_duration, or trip_length with one-way flights. One-way deal results do not contain an return_date.

What data can be retrieved from the Google Flights Deals API?

The API returns information about the departure location and a list of available flight deals.

The departure_informations object can include:

Departure airport name and IATA code
Departure city and city ID
Departure country
Airport latitude and longitude

The deals array can include:

Destination name and country
Destination Google Knowledge Graph ID
Deal price
Average price for the route
Discount percentage
Outbound and return dates
Departure and arrival airport codes
Flight duration in minutes
Number of stops
Airline name and code
Destination description and highlights
Destination thumbnail
Direct Google Flights link
SerpApi link for retrieving detailed flight information

Some fields, including average_price, discount_percentage, airline, and description, are not always present. The return_date field is also omitted for one-way flights.

Here is a simplified example of a deal:

{
  "destination_id": "/m/0k02q",
  "name": "São Vicente",
  "country": "Cape Verde",
  "price": 468,
  "average_price": 2023,
  "discount_percentage": 77,
  "outbound_date": "2026-09-21",
  "return_date": "2026-09-27",
  "departure_airport_code": "CDG",
  "arrival_airport_code": "VXE",
  "flight_duration": 350,
  "stops": 0,
  "airline": "Cabo Verde Airlines",
  "airline_code": "VR",
  "flight_link": "https://www.google.com/travel/flights?..."
}

The main results are available inside the deals array.

Scraping Google Flights

If you're interested in scraping the regular Google Flights website, we also have an API for that: Google Flights API

Here is the blog tutorial on how to scrape Google Flights.

Here is a video tutorial if you prefer to watch

https://www.youtube.com/embed/zqZSvuj7JMs?feature=oembed

Conclusion

The SerpApi Google Flights Deals API makes it possible to retrieve Google Flights deal discovery results as structured JSON.

With one request, you can collect destinations, deal prices, average prices, discount percentages, travel dates, airport codes, airlines, flight durations, stops, and links to Google Flights.

You can also customize the search using exact dates, flexible date ranges, trip durations, maximum prices, stop limits, travel classes, and airline filters.

For the complete and current parameter list, review the official Google Flights Deals API documentation.

Grok AI API tutorial - from Chat to Web Search

Hilman Ramadhan — Mon, 30 Mar 2026 02:03:37 +0000

The xAI Grok API provides access to powerful frontier models like Grok 4 series, supporting chat completions (text + vision), image generation, tool calling (function calling + built-in tools like web search), and more advanced features.

Quick Intro

Sign up at https://x.ai/api
Generate an API key from the console
Install: pip install xai-sdk
Set env var: export XAI_API_KEY="your_key_here"
Models list: https://docs.x.ai/developers/models

I'll share some samples in Python.

Basic Chat API Call

Let's first prepare our project before making the API call

Install the xai-sdk

pip install xai-sdk

Set env var: export XAI_API_KEY="your_key_here" or use .env file

Now, create a new file and this basic setup:

import os
from xai_sdk import Client
from xai_sdk.chat import user, system

from dotenv import load_dotenv
load_dotenv()
XAI_API_KEY =  os.environ.get("XAI_API_KEY")

client = Client(api_key=XAI_API_KEY)

Ensure you can print out your XAI_API_KEY correctly at this stage.

Next, let's call the chat function

...
model = "grok-4-1-fast-non-reasoning"
chat = client.chat.create(model=model)
chat.append(system("You are Grok, a highly intelligent, helpful AI assistant."))
chat.append(user("How can I be a good developer?"))

response = chat.sample()
print(response.content)

Feel free to switch the model based on your needs or preferences.

Here is an example output:

Image Generation API

Let's see how to generate an image with Grok API. We'll need to use the "grok-imagine-image" model for this.

...
response = client.image.sample(
    model="grok-imagine-image",  
    prompt="detective cat searching on website"
)

print(f"Generated image: {response.url}")

The output is a URL like this:

Video Generation API

Generating a video is as easy as generating an image with Grok API. We'll need to use the "grok-imagine-video" model for this.

response = client.video.generate(
    prompt="A glowing crystal-powered rocket launching from the red dunes of Mars, ancient alien ruins lighting up in the background as it soars into a sky full of unfamiliar constellations",
    model="grok-imagine-video",
    duration=10,
    aspect_ratio="16:9",
    resolution="720p",
)
print(response.url)

You can set the duration, aspect ratio, and resolution.

The xAI Grok API features powerful tool calling capabilities, allowing Grok to go far beyond simple text generation. It can take real actions such as performing web searches, running code, retrieving information from your own data sources, or invoking any custom functions you've defined.

Let's start by calling a custom function, as it'll help us call any internal or external API or function.

Let's say we want to call a function to look for an item's price. First, we need to define the function, such as adding the name, description, and parameters.

...
import json
from xai_sdk.chat import user, tool, tool_result
...

# Define tools
tools = [
    tool(
        name="get_item_price",
        description="Get the price of an item from the store",
        parameters={
            "type": "object",
            "properties": {
                "item_name": {"type": "string", "description": "Name of the item to get the price for"},
            },
            "required": ["item_name"]
        },
    ),
]

Upon calling the client method, we now need to include the tool we declared above.

chat = client.chat.create(
    model="grok-4.20-reasoning",
    tools=tools,
)
chat.append(user("What is the price of a laptop?"))
response = chat.sample()

print("========= response ===========")
print(response)
print("==========================")

Important: At this stage, Grok doesn't care if we have the actual function to check the price or not. The AI simply wants to know "what tools are available" for them to use.

Try to run the code to see the output from the chat call.

As you can see, Grok can detect the tool we need to call. You can see it from outputs > message > tool_calls . It consists of the name of the function and the arguments that are extracted from the user's prompt, so it'll be dynamic.

Function call simulation

Next, let's create a fake function to call. In real life, it could be a call to a database or APIs.

def get_item_price(item_name):
    prices = {
        "laptop": 999.99,
        "smartphone": 499.99,
        "headphones": 199.99,
    }
    return {"item_name": item_name, "price": prices.get(item_name, "Item not found")}

Following up from the latest code, we can check if the response has a "tool_calls" object or not. If so, we'll call the actual function we just declared above.

# Handle tool calls
if response.tool_calls:
    chat.append(response)
    for tc in response.tool_calls:
        args = json.loads(tc.function.arguments)
        result = get_item_price(args["item_name"])
        chat.append(tool_result(json.dumps(result)))


    response = chat.sample()

print(response.content)

We need to loop through the tool_calls object
We need to extract the argument to pass to the function
Call the actual function alongside the argument value
Add the information back to our chat method

Now, calling the chat.sample() method, will include all the information we received from calling the "fake function" before.

Let's try with a different prompt

chat.append(user("I need to buy two laptops and a smartphone. Can you tell me how much that will cost?"))

Here is the result

Web Search API

Grok can access real-time information through this feature, so you can get up-to-date content. Unlike the function calling above, we don't need to declare a custom function, as it's an internal tool. Here is a simple example:

import os
from xai_sdk import Client
from xai_sdk.chat import user
from xai_sdk.tools import web_search

from dotenv import load_dotenv
load_dotenv()
XAI_API_KEY =  os.environ.get("XAI_API_KEY")

client = Client(api_key=XAI_API_KEY)

chat = client.chat.create(
    model="grok-4.20-reasoning",  # reasoning model
    tools=[web_search()],
    include=["verbose_streaming"],
)

chat.append(user("Grok VS OpenAI API"))

is_thinking = True

for response, chunk in chat.stream():
    for tool_call in chunk.tool_calls:
        print(f"\nCalling tool: {tool_call.function.name} with arguments: {tool_call.function.arguments}")
    if response.usage.reasoning_tokens and is_thinking:
        print(f"\rThinking... ({response.usage.reasoning_tokens} tokens)", end="", flush=True)
    if chunk.content and is_thinking:
        print("\n\nFinal Response:")
        is_thinking = False
    if chunk.content and not is_thinking:
        print(chunk.content, end="", flush=True)

print("\n\nCitations:")
print(response.citations)

Use tools=[web_search()]
To show what's happening in the process, we use include=["verbose_streaming"],
is_thinking variable is to check if the process is still running (a boolean variable)

As you can see, it'll perform several searches on the internal with different queries. It'll then visit a specific URL after that to get more context.

Allowed domains

You can search only in specific domains using allowed_domains.

 tools=[
        web_search(allowed_domains=["grokipedia.com"]),
    ],

Exclude Domains

Vice versa, you can exclude specific domains

chat = client.chat.create(
    model="grok-4.20-reasoning",
    tools=[
        web_search(excluded_domains=["grokipedia.com"]),
    ],
)

Better Web Search API

While you can specifically choose the domain, the keyword Grok uses to find answers on the internet is random. For example, when I'm asking for "Top 3 pizza restaurants from Google Maps in Boston. Share some reviews and ratings for each place."

This is what I saw from the thinking process:

It needs to perform multiple queries before returning the answer.

Another sample, when asking simply for 3 images:

It runs on multiple pages, and unfortunately, the links are not valid. Grok may hallucinate at this point.

Web Search API alternative

In some cases, AI-generated keywords are fine, but if you're building an app where you want efficiency and full control of the process, the native "Web Search Tool" can be replaced with a simple API call to a specific API your app needs. For example, to find answers on the internet, SerpApi provides 100+ APIs for you.

Need a generic Google answer? We have:

- Google Search API

- Google AI Overview

- Google AI Mode

and more!

See how SerpApi is the Web Search API for your AI apps, LLM, and agents.

Using Grok API with SerpApi

To get some idea of how SerpApi works, feel free to test the results on our playground. You can play with different parameters and directly see the JSON sample we return.

Sample case

Let's say we want to find images via Google Image API like this:

Step 1: Preparation

You can register for free at serpapi.com to get your API key.

Step 2: Parsing keyword

Let's say we need 3 images from Google. Since users can type anything, we need to parse the keyword, as SerpApi simply performs a search using a particular keyword.

USER_QUERY = "Show me 3 cute cat images from the internet"

# Step 1: Ask Grok to extract a search keyword from the user's natural language
keyword_chat = client.chat.create(model="grok-3-fast")
keyword_chat.append(system("Extract the most relevant search keyword or phrase from the user's message. Reply with only the keyword, nothing else."))
keyword_chat.append(user(USER_QUERY))

keyword_response = keyword_chat.sample()
search_keyword = keyword_response.content.strip()
print(f"Extracted keyword: {search_keyword}")

Step 3: Search via SerpApi

We now have the keyword. Let's run a search on SerpApi

# Step 2: Search via SerpAPI using simple requests (Google Images)
serpapi_params = {
    "api_key": SERPAPI_API_KEY,
    "engine": "google_images",
    "q": search_keyword,
    "hl": "en",
    "gl": "us",
}

serpapi_url = "https://serpapi.com/search"
serpapi_response = requests.get(serpapi_url, params=serpapi_params)
results = serpapi_response.json()

At this stage, you already have the answers you're looking for.

Step 4: (Optional) Filter results

Sometimes, we don't need all the information. It's good to filter it programmatically first, so we don't use too many tokens.

For example, I'm only interested in the top 5 answers:

image_results = results.get("images_results", [])
[:5]
formatted_results = "\n".join(
    f"- {img.get('title', 'No title')}: {img.get('original', img.get('thumbnail', 'No URL'))}"
    for img in image_results
)
print(f"\nSerpAPI results:\n{formatted_results}")

We can also format the answer as a bonus.

Step 5: (Optional) Reply in a natural language

Depending on your application, you may want to answer the user back in natural language. We just need to pass the answers above back to the AI:

# Step 3: Feed results back to Grok for a final response
final_chat = client.chat.create(model="grok-3-fast")
final_chat.append(system("You are a helpful assistant. Use the provided search results to answer the user's question."))
final_chat.append(user(f"User question: {USER_QUERY}\n\nSearch results from SerpAPI:\n{formatted_results}\n\nPlease answer the user's question based on these results."))

final_response = final_chat.sample()
print(f"\nFinal Response:\n{final_response.content}")

Final result:

Sidenote

It's also possible to call the API with the OpenAI SDK. Sample:

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("XAI_API_KEY"),
    base_url="https://api.x.ai/v1",
)

How to scrape 100 search results on Google (num parameter)

Hilman Ramadhan — Thu, 25 Sep 2025 01:48:54 +0000

Finally, there is a solution to scrape 100 search results from Google after they dropped the num parameter: introducing the Google Fast Light API

Google recently discontinued support for showing up to 100 results per page in search, a parameter that many SEO professionals, data companies, and researchers have relied on for years. This change disrupts established workflows, making large-scale data collection, competitor analysis, and keyword research less efficient.

API to scrape 100 search results from Google

Introducing Google Light Fast API

With this API, you can scrape 100 search results from Google with one single request. This API only contains the organic_results , which makes it faster compared to our regular Google Search API.

You can access this API with a single GET request:

https://serpapi.com/search.json?engine=google_light_fast&q=coffee&api_key=YOUR_API_KEY&num=100

You can replace coffee with any keyword you want to search for.
To get your API key, you can register for free at SerpApi.

Scraping other rich results from Google

If you need the complete Google results on the first page, you can still use our Google Search API in order to scrape AI Overview answer, knowledge graph, and so on. So, if you need both 100 organic results and complete rich results , you only need to perform two searches:

1 to get the 100 organic results via the new Google Fast Light API
2 to get the rich results on the Google Search API.

No need to paginate the results or run 10 searches anymore.

Conclusion

The removal of the num=100 (&num=100) parameter from Google has a lot of impacts on many SEO companies that need to serve ranking data for their clients. Many search API providers fail to scrape 100 search results, which results in many errors and misinformation on their platform.

This issue was also mentioned a couple of times on X (Twitter) by many search engine professionals,

Barry Schwartz - founder of the Search Engine Roundtable

Brodie Clark - Independent SEO consultant

SEO companies looking for a way to track their clients’ SEO rankings programmatically, as well as AI companies seeking more real-time data from Google, should try this new API.

How to scrape Tripadvisor (2025 Tutorial)

Hilman Ramadhan — Fri, 19 Sep 2025 02:17:42 +0000

Scraping Tripadvisor listings allows you to gather detailed information about various attractions, hotels, or restaurants, including descriptions, amenities, pricing, and user-generated ratings. This data can be instrumental in understanding market trends, identifying competitors, and optimizing your own listings or offerings in the travel industry.

Luckily, you can do this easily using our brand new Tripadvisor API. We'll see how to scrape it in cURL, Python, and JavaScript. We can scrape restaurants, things to do, hotels, destinations, vacation rentals, and forum information.

Available data on the Tripadvisor Search API

Here is the list of data you can retrieve from this Tripadvisor Search API:

title
description
rating
reviews
location
thumbnail
highlighted overview

This is perfect if you need to collect place data from Tripadvisor.

How to scrape the Tripadvisor website?

Now, let's see how to use this simple API from SerpApi to collect the data!

Get your API Key

First, ensure you register at serpapi.com to get your API Key. You can get 250 free searches per month. You can use this API Key to access all of our APIs, including the Tripadvisor Search API.

Available parameters

On top of running the basic search, you can see all of our available parameters here.

cURL Implementation

Here is the basic implementation in cURL:

curl --get https://serpapi.com/search \
 -d api_key="YOUR_API_KEY" \
 -d engine="tripadvisor" \
 -d q="Rome"

The q parameter is responsible for the search query.

Sample response from cURL request

Scrape Tripadvisor search results in Python

Next, let's see how to scrape the Tripadvisor search results in Python.

Preparation for accessing the SerpApi API in Python

Create a new main.py file
Install requests with:

pip install requests

Here is what the basic setup looks like:

import requests
SERPAPI_API_KEY = "YOUR_REAL_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, #replace with your real API Key
    # soon
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

With these few lines of code, we can access all of the search engines available at SerpApi, including the Tripadvisor Search API.

import requests
SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "tripadvisor",
    "q": "indonesia"
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

To make it easier to see the response, let's add indentation to the output.

import json

# ...
# ...
# all previous code

print(json.dumps(response, indent=2))

Running this Python file should show you the listing for that keyword from Tripadvisor website.

Print specific information

Let's say we only need the title, description, rating, and reviews. This is how we can print specific columns from the locations results:

# ...
search = requests.get("https://serpapi.com/search", params=params)
response = search.json()

for result in response.get("locations", []):
    title = result.get("title")
    rating = result.get("rating")
    reviews = result.get("reviews")
    description = result.get("description")

    print(f"Title: {title}")
    print(f"Rating: {rating}")
    print(f"Reviews: {reviews}")
    print(f"description: {description}")
    print("-" * 10)

Export data to a CSV file

Let's see how to export this Tripadvisor product data into a CSV file in Python

# ...

import csv

with open("tripadvisor_results.csv", "w", newline="", encoding="utf-8") as csvfile:
    fieldnames = ["title", "rating", "reviews", "description"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    for result in response.get("locations", []):
        writer.writerow({
            "title": result.get("title"),
            "rating": result.get("rating"),
            "reviews": result.get("reviews"),
            "description": result.get("description")
        })
print("Data exported successfully.")

JavaScript implementation

Finally, let's see how to scrape the Tripadvisor search results in JavaScript.

Install the serpapi package:

npm install serpapi

Run a basic query:

const { getJson } = require("serpapi");
getJson({
  engine: "tripadvisor",
  api_key: API_KEY, // Put your API Key
  q: "indonesia"
}, (json) => {
  console.log(json["locations"]);
});

Other programming languages

While you can use our APIs using a simple GET request with any programming language, you can also see our ready-to-use libraries here: SerpApi Integrations.

How to customize the search?

We provide many filters to customize your search.

tripadvisor_domain: Parameter defines the Tripadvisor domain to use. It defaults to tripadvisor.com. Head to Tripadvisor domains for a full list of supported domains.
lat and lon: Specify the lat and lon GPS coordinates when performing the search

Example using a different Tripadvisor domain:

curl --get https://serpapi.com/search \
 -d api_key="YOUR_API_KEY" \
 -d engine="tripadvisor" \
 -d q="cafe" \
 -d tripadvisor_domain="wwww.tripadvisor.ca"

Advanced Parameters:

ssrc: This parameter specifies the search filter you want to use for the Tripadvisor search.

Available options:

a - All Results

r - Restaurants

A - Things to Do

h - Hotels

g - Destinations

v - Vacation Rentals

f - Forums

How to paginate the results?

You can scrape beyond the first page using the offset and limit parameter.

limit defines how many results you want to receive per page. By default, it's 30. Maximum value is 100.
Offset skips the given number of results. For example, 0 for the first page, 30 for 2nd page, 60 for 3rd page, and so on. Or if you're using 100 as the limit, then the 2nd page will be 100, 3rd page will be 200, and so on.

Frequently Asked Questions (FAQs)

Is it legal to scrape the Tripadvisor website?

Scraping publicly available data from websites like Tripadvisor is generally permitted under U.S. law.

How much does it cost?

Register at serpapi.com to start for free. If you want to scale, we offer tiered plans based on your usage.

Why do you need to scrape Tripadvisor listing?

Scraping Tripadvisor listings can provide valuable data for market analysis, competitive insights, and understanding customer preferences in the travel and hospitality industry. This information can help businesses optimize their offerings and improve customer engagement.

Closing

That's it! Thank you very much for reading this blog post. You can play around for free on our playground here.

Scrape YouTube videos in Python

Hilman Ramadhan — Fri, 08 Aug 2025 08:10:37 +0000

Scraping YouTube videos enables developers and businesses to extract detailed YouTube video metadata at scale, including titles, descriptions, view counts, thumbnails, channel names, related videos, and comments/replies. It streamlines what would otherwise require complex scraping and anti‑blocking measures.

Get your API Key

First, ensure to register at SerpApi to get your API Key. You can get 250 free searches per month. Use this API Key to access all of our APIs, including the YouTube Video API.

Available parameters

In addition to running the basic search, you can view all YouTube Video API parameters here.

How to scrape YouTube video data with Python

Create a new main.py file
Install requests with:

pip install requests

Here is what the basic setup looks like:

import requests
SERPAPI_API_KEY = "YOUR_REAL_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, #replace with the actual API Key
    # soon
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

With these few lines of code, we can access all of the search engines available at SerpApi, including the YouTube Video API.

import requests
SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "youtube_video",
    "v": "j3YXfsMPKjQ" # YouTube video ID
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

To make it easier to see the response, let's add indentation.

import json

# ...
# ...
# all previous code

print(json.dumps(response, indent=2))

Here is the result:

Here is how to scrape the comments on a video.

From the request we performed previously, you should be able to see this in the response.

We can scrape the "Top comments" or "Newest first" comments using the token that is available for each. We can put this token in the next_page_token parameter.

Here is an example:

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "youtube_video",
    "v": "j3YXfsMPKjQ", # YouTube video ID
    "next_page_token": "Eg0SC2ozWVhmc01QS2pRGAYyOCIRIgtqM1lYZnNNUEtqUTABeAIwAUIhZW5nYWdlbWVudC1wYW5lbC1jb21tZW50cy1zZWN0aW9u"
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(json.dumps(response, indent=2))

Here is the result:

If you're interested in scraping the reply on the comment, you can repeat the same action, but this time using the replies_next_page_token.

Bonus

You can also scrape YouTube search results using our YouTube Search API. Here is how to scrape YouTube search results using Python.

Why Use It?

Real‑time data: Fetch up‑to‑date video details.
Rich metadata access: Retrieve comments and nested replies, related videos, and viewer stats in one request.
Automated proxy and captcha handling: SerpApi manages the complexities of scraping so you can focus on analysis.

Key Features

Feature: Comprehensive video metadata
- Benefit: Includes title, description, duration, chapters, views, publishing date, channel, thumbnails
Feature: Comments & replies
- Benefit: Access nested comments for engagement analysis
Feature: Related videos
- Benefit: Discover context and competition around a video

Build a SERP rank tracker app with this API

Hilman Ramadhan — Mon, 23 Sep 2024 07:48:06 +0000

If you're interested in building a SERP ranking tracker app, you'll love our API. SerpApi provides a simple API to access live data from various search engines, including Google, Bing, DuckDuckGo, Yahoo, and others. It enables you to build an app like a SERP ranking tracker.

The idea

To get the ranking position of a website, we need to access the organic results and check where the domain first appears. The organic results data is available through our API. Here are the three APIs we're going to use:

Google Search API: Scrape the results from Google search.
Bing Search API: Scrape the results from the Bing search.
DuckDuckGo Search API: Scrape the results from the DuckDuckGo search.

API Design

We'll create a single endpoint where people can receive the ranking results from the above search engine using this parameter:

Domain name (String): The website domain we want to track.
Keywords (Array): List of keywords that we want to search for.
Engines (Object): List of search engines we want to search on. We can also adjust the parameter details based on the API. Refer to the relevant documentation to check the available parameters for each APIs.

POST: /api/rankings

Data:
 - domain (string)
 - keywords (array[string])
 - engines ((array[name, params]))

Example:
{
  "domain": "archive.org",
  "keywords": ["internet archive"],
  "engines": [
    {
      "name": "google",
      "params": {
        "domain": "google.com",
        "gl": "es"
      }
    }
  ]
}

The source code is available on GitHub; feel free to take a look at the detailed implementation here:
https://github.com/hilmanski/rank-tracker-api/

Let's write the code!

I'll use Nodejs for this API; feel free to use other languages/frameworks.

Install Express

Let's use Express to help us clean up the code structure.

npm i express --save

Export your API Key

You can either export your API key in a terminal like the sample below or save it on an .env file.

export SERPAPI_API_KEY=YOUR_ACTUAL_API_KEY

Basic route

Prepare the POST endpoint with relevant parameters

const express = require('express')
const app = express()
const port = 3000

app.use(express.json());

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

  // detail implementation later

  res.json({ keywords, engines });
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Validate the input type

Make sure API users use the correct types.

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

   // Validate keywords
  if (!Array.isArray(keywords) || !keywords.length) {
    return res.status(400).json({ error: 'Keywords and engines must be arrays.' });
  }

  // Validate engines
  for (const engine of engines) {
    if (typeof engine !== 'object' || !engine.name) {
      return res.status(400).json({ error: 'Each engine must be an object with a "name" property.' });
    }
    if (engine.params && typeof engine.params !== 'object') {
      return res.status(400).json({ error: 'Engine "params" must be an object.' });
    }
  }

  // coming soon

})

Run parallel search

Since we're enabling multiple keywords and multiple search engines, we need to run the function in parallel to save us some time.

// Parallel search
  const results = await Promise.all(engines.map(async engine => {
    const rankings = await Promise.all(keywords.map(async keyword => {
      return await getRanking(keyword, engine, cleanDomain);
    }));

    // map keywords - rankings in one array
    const rankingResults = keywords.map((keyword, index) => {
      return [keyword, rankings[index]];
    });

    console.log(rankingResults);

    return { domain, engine, rankingResults };
  }))

*getRanking method coming soon.

GetRanking method implementation

Here is the function that is responsible for running the search for each search engine.

const suportEngines = ['google', 'bing', 'duckduckgo'];

async function getRanking(keyword, engine, domain) {
  const engineName = engine.name.toLowerCase();

  if(!suportEngines.includes(engineName)) {
      console.error(`Error: Engine ${engineName} is not supported.`);
      return;
  }

  return new Promise(async (resolve, reject) => {
      switch(engineName) {
          case 'google':
            resolve(await searchGoogle(keyword, engine.params, domain))
          break;
          case 'bing':
            resolve(await searchBing(keyword, engine.params, domain))
          break;
          case 'duckduckgo':
            resolve(await searchDuckDuckGo(keyword, engine.params, domain))
          break;
          default:
          break;
      }
  })
}

We'll use the native fetch method in NodeJS to request the actual SerpApi endpoint for each search engine.

API to access ranking position in Google

function searchGoogle(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=google&num=100&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

API to access ranking position in Bing

function searchBing(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=bing&count=50&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

API to access ranking position in DuckDuckGo

function searchDuckDuckGo(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=duckduckgo&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

Final Endpoint

Here's what our endpoint looks like

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

  // Validate keywords
  if (!Array.isArray(keywords) || !keywords.length) {
    return res.status(400).json({ error: 'Keywords and engines must be arrays.' });
  }

  // Validate engines
  for (const engine of engines) {
    if (typeof engine !== 'object' || !engine.name) {
      return res.status(400).json({ error: 'Each engine must be an object with a "name" property.' });
    }
    if (engine.params && typeof engine.params !== 'object') {
      return res.status(400).json({ error: 'Engine "params" must be an object.' });
    }
  }

  // CLean up domain
  // Since people can include https:// or http:// or a subdomain, strip all of it?
  const cleanDomain = domain.replace(/^https?:\/\//, '').replace(/\/$/, '');

  // Parallel search
  const results = await Promise.all(engines.map(async engine => {
    const rankings = await Promise.all(keywords.map(async keyword => {
      return await getRanking(keyword, engine, cleanDomain);
    }));

    // map keywords - rankings in one array
    const rankingResults = keywords.map((keyword, index) => {
      return [keyword, rankings[index]];
    });

    console.log(rankingResults);

    return { domain, engine, rankingResults }
  }))

  res.json(results);
})

Test the API

Let's try out this API via cURL.

No parameter example

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
  -d '{
    "keywords": ["internet archive"],
    "domain": "archive.org",
    "engines": [
      {
       "name": "google"
     }
    ]
  }'

Using multiple keywords and search engine parameter sample

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
-d '{
    "keywords": ["internet archive", "digital library archived internet"],
    "domain": "archive.org",
    "engines": [
      {
        "name": "google",
        "params": {
            "google_domain": "google.co.id",
            "gl": "id"
        }
      }
    ]
  }

Using multiple search engines

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
  -d '{
    "keywords": ["internet archive", "digital archive", "internet library"],
    "domain": "archive.org",
    "engines": [
    {
      "name": "google",
      "params": {
        "domain": "google.com",
        "gl": "es"
      }
    },
    {
      "name": "Bing",
      "params": {
        "cc": "gb"
      }
    },
{
      "name": "duckduckgo",
      "params": {
        "kl": "uk-en"
      }
    }
  ]
  }'

That's it! I hope you like this post. Please let us know if you have any questions. Feel free to also contribute to this project on GitHub.

Create a super fast AI assistant with Groq (Without a database)

Hilman Ramadhan — Thu, 16 May 2024 02:47:59 +0000

Last week, I tried to build a voice AI assistant using OpenAI AI assistant. It takes a while to generate a response, which is not suitable for a voice assistant. So, I'm looking for an alternative to make my assistant faster. That's how I found out about Groq. This post will cover how I build an AI assistant using Groq.

Pros and Cons summary

Pro:

Easy to implement with only one API (Groq API).

Respond is fast.

Cons:

The longer we chat, the higher the chance that we might lose some context along the way.

What is Groq?

Groq is a service that provides a super fast engine to run AI applications. It's not an AI model! We can run different AI models like Llama, Mixtral, Gemma and more!

Ref: Why Groq?

How I build a fast AI assistant

Many AI models exist, but only OpenAI offers an easy way to implement a chat-like experience using the Assistants API. By default, these models won't know or understand the context of our previous chat. So, we have to re-explain everything if we want the AI to understand the context of each message.

There are some alternatives out there, such as using LangChain chat history. But I prefer to find a simple way (*with the caveat, of course). Luckily, I found some ideas on the internet (Thank you, Internet!).

The idea below can be implemented for any AI model/engine, not just Groq. You can try this with OpenAI itself, Mixtral, Claude, and so on.

Here is the flow:

The user sends the initial message
The AI responds to the message
We ask AI to summarize the conversation
We send the response and summary back to the user
The user will send the summary back later alongside the new message
AI now will reply based on the fresh message and with help of the conversation summary to provide some context.

The caveat of this method

By summarizing a conversation, we may lose some information along the way. That's why it's a good idea in certain cases to store the message history on a database (Vector database).

One way I can reduce this shortage is by attaching the recent reply from AI. I've also read an article that suggests keeping the latest 2-3 conversations and providing them as additional context later.

Code implementation

I'll use NodeJs for this tutorial. Feel free to use any language you want. The final code is available at GitHub:

GitHub -assistants-api-with-groq-ai

Install dependencies

npm i express groq-sdk dotenv --save

Express for creating a route for the endpoint
Groq-sdk is the official package for using Groq in Javascript
dotenv to store our API key safely.

Add API Key

Create a new .env file. Add your Groq API key in this file like this:

GROQ_API_KEY=YOUR_GROQ_API_KEY

Make sure to sign up to Groq and get your API key here.

Basic Setup

Let's create a new index.jsfile, and we'll write everything in this file. We prepare one endpoint called chat where we'll send these parameters:

- message: user's message

- latestReply: The latest reply from AI

- messageSummary: The conversation summary so far

In this endpoint, we'll do two things:

- Respond to new user message (with latestReply and messageSummary as context)

- Create a new conversation summary by providing the fresh reply from AI.

const express = require('express');

// Express Setup
const app = express();
app.use(express.json());
const port = 3000

require("dotenv").config();
const { GROQ_API_KEY } = process.env;

// GROQ Setup
const Groq = require("groq-sdk");
const groq = new Groq({
    apiKey: GROQ_API_KEY
});

async function chatWithGroq() { } // soon
async function summarizeConversation() { } // soon

app.post('/chat', async (req, res) => {
    const { message, latestReply, messageSummary } = req.body;

    // request chat completion
    const reply = await chatWithGroq(message, latestReply, messageSummary)

    // request chat summary
    const summary = await summarizeConversation(message, reply, messageSummary)

    // Always return chat history/summary
    res.send({
        reply,
        summary
    })
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Chat with Groq method

Here is the chatWithGroq method implementation:

async function chatWithGroq(userMessage, latestReply, messageHistory) {
    let messages = [{
        role: "user",
        content: userMessage
    }]

    if(messageHistory != '') {
        messages.unshift({
            role: "system",
            content: `Our conversation's summary so far: """${messageHistory}""". 
                     And this is the latest reply from you """${latestReply}"""`
        })
    }

    console.log('original message', messages)

    const chatCompletion = await groq.chat.completions.create({
        messages,
        model: "llama3-8b-8192"
    });

    const respond = chatCompletion.choices[0]?.message?.content || ""
    return respond
}

We only provide a conversation summary when we have one (look at the if statement). So, it won't be included in our first message.

Conversation summary method

Here is the summarizeConversation method implementation:

async function summarizeConversation(message, reply, messageSummary) {
    let content = `Summarize this conversation 
                    user: """${message}""",
                    you(AI): """${reply}"""
                  `

    // For N+1 message
    if(messageSummary != '') {
        content = `Summarize this conversation: """${messageSummary}"""
                    and last conversation: 
                    user: """${message}""",
                    you(AI): """${reply}"""
                `
    }

    const chatCompletion = await groq.chat.completions.create({
        messages: [
            {
                role: "user",
                content: content
            }
        ],
        model: "llama3-8b-8192"
    });

    const summary = chatCompletion.choices[0]?.message?.content || ""
    console.log('summary: ', summary)
    return summary
}

In this method, we ask the AI to create a summary based on the latest summary and recent reply.

Demo Time!

You can use any API client, like Postman, Thunder (VS Code), etc.

Don't forget to run your program with node index.js

Create a POST request for the /chat endpoint and provide message endpoint and provide the first message parameter.

We can display the reply from the response on our user interface. This is the actual reply to our message.

We'll save the summary for the next request.

Now, this is how the JSON looks like for the N+1 message:

The next messages should include the latestReply and messageSummary as parameters.

message: *Don't forget to add a new message. This is you talking to the AI. Notice that I use here on my question, to validate that the AI knows what's the previous context here.
latestReply: Send the latest reply from AI (from previous response)
messageSummary: Send the conversation summary so far (from previous response)

Here is the result to this request:

As you can see, the AI knows that when I said here I was talking about Indonesia. You can try to send a follow-up message (create a new request) by asking something like "Can you tell me more about number 4?" as an example. But don't forget that we always need to update the latestReply and summaryConversation on each request.

To return the response and summarize the conversation, I only need to wait around 2s. This is much faster than using OpenAI AI assistants.

Reference:

- Build a smart AI voice assistant

- Basic tutorial: Assistants API by OpenAI

Build an AI Voice assistant like Siri (use OpenAI AI Assistant)

Hilman Ramadhan — Tue, 30 Apr 2024 00:23:48 +0000

Hi! Today, we'll learn how to build an AI Voice assistant like Siri that can understand what we say and speak back to us.

Demo AI Voice assistant

Here's what we're going to build:

This post will focus on implementing it as a web application. Therefore, we will use HTML for the interface and Javascript for the voice features. You might want to adjust this if you build for another platform (mobile, desktop, etc.).

Regardless of the platform, knowing the components of how to build this will help us along the way.

Basic Voice AI assistant Structure

Just like other applications, we will have this basic structure:

Here is what it looks like on our app:

You can replace each of these elements with the more advanced option. I'm trying to stick with what we already have in the browser. These are the alternatives:

Voice input: AssemblyAI or OpenAI Whisper
Voice output: Elevenlabs or OpenAI Whisper

Code Tutorial on how to build a Voice AI assistant

We'll split our codebase into two parts, one for the front end and one for the back end.

First, we just want to make sure that we can get a text from the user's voice and read a text out loud; there is no AI or conversation involved yet.

Here is the final code result of this post:
GitHub - hilmanski/simple-ai-voice-assistant-openai-demo

Step 1: Basic frontend Views

Since we're building a web application, we'll use HTML.

- We need two buttons to start and stop the recording

- A div to display the text

<button id="record">Record</button>
<button id="stop">Stop</button>

<div id="output">Output</div>

(Optional) If you want to copy the style I implemented:

<style>
body {
    margin: 50px auto;
    width: 500px;
}

#output {
    margin-top: 20px;
    border: 1px solid #000;
    padding: 10px;
    height: 200px;
    overflow-y: scroll;
}

#output p:nth-child(even) {
    background-color: #f8f6b1;
}
</style>

Step 2: Listen to speech

Let's trigger the actions from Javascript:

<script>
    // Set up
    const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition;
    const SpeechGrammarList = window.SpeechGrammarList || window.webkitSpeechGrammarList;
    const SpeechRecognitionEvent = window.SpeechRecognitionEvent || window.webkitSpeechRecognitionEvent;

    const recognition = new SpeechRecognition();
    const speechRecognitionList = new SpeechGrammarList();

    recognition.grammars = speechRecognitionList;
    recognition.continuous = true;
    recognition.lang = 'en-US';
    recognition.interimResults = false;
    recognition.maxAlternatives = 1;


    // Start recording
    document.getElementById('record').onclick = function() {
        recognition.start();
    }

    // Stop recording
    document.getElementById('stop').onclick = function() {
        recognition.stop();
        console.log('Stopped recording.');
    }

    // Output
    recognition.onresult = async function(event) {
        // Get the latest transcript 
        const lastItem = event.results[event.results.length - 1]
        const transcript = lastItem[0].transcript;
        document.getElementById('output').textContent = transcript;
        recognition.stop(); 
        // await sendMessage(transcript); // we'll implement this later
    }

    recognition.onspeechend = function() {
        recognition.stop();
    }
</script>

Step 3: Backend structure

We're using NodeJS/Express for the backend. Make sure to install the necessary packages in your new directory (separate from the frontend code):

npm init -y #initialize NPM package
npm init express cors dotenv openai --save
touch index.js #create a new empty file

const express = require('express');
const cors = require('cors')

// Setup Express and allow CORS
const app = express();
app.use(express.json());
app.use(cors()) // allow CORS for all origins

// Main route
app.post('/message', async (req, res) => {
    const { message } = req.body;
    res.json({ message: 'Received: ' + message });
});

// Start the server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Run node js in your terminal to run the server. Your application is running on localhost:3000.

We only have one route, which is /message , which receives a message from the client and echoes it back. This is to ensure that the input and output parts are running smoothly.

Step 4: Send a message from frontend

Let's add a fetch method to send the message to that endpoint.

Update your recognition.onresult event to call the sendMessage function like this:

recognition.onresult = async function(event) {
    // Get the latest transcript 
    const lastItem = event.results[event.results.length - 1]
    const transcript = lastItem[0].transcript;
    document.getElementById('output').textContent = transcript;

    await sendMessage(transcript); // New addition
}

Now, let's declare the sendMessage method:

async function sendMessage(message) {
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message })
    });

    const data = await response.json();
    console.log(data);

    speak(data.message);
}

function speak(message) {
    if (synthesis) {
        const utterance = new SpeechSynthesisUtterance(message);
        synthesis.speak(utterance);
    }
}

I also added a new method called speak to use the Web Speech API, specifically SpeechSynthesis method to speak. The SendMessage will hit the endpoint we prepared previously.

Step 5: Try the App

Now, run your HTML file, you can use something like VS Code Live server.

press record
say anything

AI Voice assistant web application screenshot

Now, you should see your message echo back to you (it's coming from the server). Press the record button again to send a different message.

Now, we have an app that can listen and speak to us. Let's dive into the AI part!

Smart AI assistant

We'll use the OpenAI Assistants API as the brain.

Step 1: Install the OpenAI package

We've previously installed the OpenAI package for NodeJS. Make sure you've installed it as well.

Step 2: Grab your API Key

Get your API Key from the OpenAI dashboard. Create a new .env file and paste it there.

OPENAI_API_KEY=YOUR_API_KEY_FROM_OPENAI

Step 3: Import OpenAI and dotenv

require("dotenv").config();
const OpenAI = require('openai');
const { OPENAI_API_KEY } = process.env;

// Set up OpenAI Client
const openai = new OpenAI({
    apiKey: OPENAI_API_KEY,
});

Step 4: Implement assistants API

If you're not familiar with the assistants API, I suggest to read this introduction blog post first:

[

Assistant API by OpenAI (Basic Tutorial)

Learn the basics of Assistant API by OpenAI. We’ll create a super simple coding example to understand the barebone of Assistant API: https://serpapi.com/blog/assistant-api-openai-beginner-tutorial/

We'll use the same logic and code for this tutorial (with some updates soon). You can also get the code sample for the API assistants here:

GitHub - hilmanski/assistant-API-openai-nodejs-sample: A simple example for Assistant API by OpenAI using NodeJS

Note: I won't explain and show the whole code here.

Step 5: Add the assistants API

We need to tell the AI assistants that they will act as general helpers who can help us with anything.

*Make sure to update the assistant_id key on your code.

Step 6: Assign a thread ID

Since the API needs a thread unique ID for each conversation, I'll add a new fetch request on our frontend to automatically ask for a thread id on the first visit. The ID will also updated on browser refresh.

Reminder: we have this route to create a thread ID from the OpenAI assistant tutorial

// Open a new thread
app.get('/thread', (req, res) => {
    createThread().then(thread => {
        res.json({ threadId: thread.id });
    });
})

Let's add this block to our HTML file (frontend part)

let threadId = null;

// onload
window.onload = function() {
    fetch('http://localhost:3000/thread')
        .then(response => response.json())
        .then(data => {
            console.log(data);
            assistant_id = data.threadId;
        });
}

We'll adjust our sendMessage method to attach the threadId when sending a message.

 async function sendMessage(message) {
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message, threadId }) // <- update here
    });

    // continue

Step 7: Return the last message only

Let's update our checkingStatus method to only return the latest message from the AI

async function checkingStatus(res, threadId, runId) {
    const runObject = await openai.beta.threads.runs.retrieve(
        threadId,
        runId
    );

    const status = runObject.status;
    console.log(runObject)
    console.log('Current status: ' + status);

    if(status == 'completed') {
        clearInterval(pollingInterval);

        const messagesList = await openai.beta.threads.messages.list(threadId);
        const lastMessage = messagesList.body.data[0].content[0].text.value

        res.json({ message: lastMessage });
    }
}

Our voice assistant is ready!

Step 8: Show all messages

If you want to add all previous transcriptions to the user interface, here is the code to collect all the previous messages in the div.

First, every time we speak:

recognition.onresult = async function(event) {
    // Get the latest transcript 
    const lastItem = event.results[event.results.length - 1]
    const transcript = lastItem[0].transcript;

    // Update: Append new text to div
    const newText = "<p>" + transcript + "</p>";
    document.getElementById('output').insertAdjacentHTML("afterbegin", newText);

    recognition.stop();
    await sendMessage(transcript);
}

Second, every time we got a response from the AI:

async function sendMessage(message) {
    console.log('Sending message: ', threadId);
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message, threadId })
    });

    const data = await response.json();

    // update: add new text here
    const newText = "<p>" + data.message + "</p>";
    document.getElementById('output').insertAdjacentHTML("afterbegin", newText);

    speak(data.message);
}

Make sure to re-run your NodeJS app. Now try to have a chat with your AI!

Things we can improve

There are several things we can improve for this voice assistant

AI Instruction

Since we're building a voice assistant, it shouldn't explain things too long for a simple question. We can adjust the instructions like this:

You're a general AI assistant that can help with anything. You're a voice assistant, so don't speak too much, make it clear and concise.

Voice and listening

We use native browser API to listen and speak. Better alternatives exist, such as Elevenlabs, AssemblyAI, and more.

Knowledge Limitation

We're using one of the OpenAI models, where the knowledge is cut off at a particular year. We can expand its knowledge by providing PDF files or connecting the assistant API to the internet.