The Problem: Why AI Agents Need a New Web Index
Traditional search engines like Google and Bing were designed for human users who cannot process entire webpages at once. These engines rely on keyword matching, ranking algorithms, and snippet-based results to deliver concise answers. However, AI agents—such as chatbots, virtual assistants, and autonomous research tools—operate differently. They require access to raw, structured, and comprehensive data to generate accurate, context-aware responses.
The limitations of existing search infrastructure for AI are becoming increasingly apparent. AI agents need to:
- Process vast amounts of data in real time, far beyond what a human can digest.
- Understand context across multiple pages, not just surface-level snippets.
- Retrieve structured data (e.g., tables, APIs, or metadata) rather than unstructured text.
- Avoid bias introduced by traditional ranking algorithms, which prioritize human-readable content over machine-readable data.
Andrey Styskin, co-founder of Keenable and former head of Yandex’s search, AI, and cloud division, highlights this gap: "Search engines were built for people who can’t scan entire pages. AI agents don’t have that limitation—they need a different kind of infrastructure." This insight underscores the need for a web index purpose-built for AI, not humans.
Keenable’s Solution: A Web Index for Machines
Keenable is addressing this challenge by building a web index specifically designed for AI agents. Unlike traditional search engines, which prioritize human-readable snippets, Keenable’s index focuses on:
- Raw data retrieval: Providing AI agents with direct access to unfiltered, structured web content.
- Contextual understanding: Enabling agents to process information across multiple sources without losing nuance.
- Scalability: Supporting the massive computational demands of AI-driven queries.
Technical Breakdown: How Keenable’s Index Works
While Keenable has not disclosed all technical details, its approach likely involves:
-
Crawling and Indexing:
- A distributed crawler that fetches web content at scale, similar to Googlebot but optimized for AI consumption.
- Indexing not just text but also metadata, APIs, and structured data (e.g., JSON-LD, Schema.org markup).
-
Data Structuring:
- Converting unstructured web content into machine-readable formats (e.g., knowledge graphs, embeddings, or vector databases).
- Storing data in a way that allows AI agents to retrieve and synthesize information efficiently.
-
Query Processing:
- A query engine that translates AI agent requests into optimized searches across the index.
- Support for complex, multi-step queries (e.g., "Summarize the latest research on quantum computing and compare it to 2025 trends").
-
Integration with AI Models:
- APIs that allow AI agents to plug directly into Keenable’s index, bypassing the need for traditional search engine APIs.
Partnerships and Early Use Cases
Keenable has already partnered with Gradium, a voice-AI company, to enable live information retrieval for voice-based AI assistants. This collaboration demonstrates how Keenable’s index can power real-time, context-aware responses for AI-driven applications. Other potential use cases include:
- Autonomous research tools for academia and enterprise.
- AI-powered customer support agents that retrieve and synthesize information from multiple sources.
- Enterprise knowledge management systems that index internal and external data for AI-driven insights.
Why This Matters: The AI Search Revolution
The emergence of Keenable reflects a broader shift in the search industry. Traditional search engines are struggling to adapt to the demands of AI agents, and their reluctance to open up their infrastructure is creating opportunities for startups like Keenable.
The Incumbent Problem: Google and Microsoft’s Dilemma
Google and Microsoft have historically dominated the search market, but their business models are built around human users. Opening up their search APIs to AI agents could:
- Cannibalize ad revenue: AI agents don’t click on ads, which are a primary revenue source for search engines.
- Increase computational costs: Serving AI agents requires more resources than serving human users.
- Expose proprietary data: AI agents could extract and synthesize data in ways that undermine search engines’ control over information.
As a result, both companies have shut down or restricted access to their search APIs, leaving a gap in the market. This has created an opening for startups like Keenable, Brave, and Exa to build alternative infrastructure.
The Competitive Landscape
Keenable is not alone in this space. Other players are also exploring AI-first search solutions:
- Brave: Known for its privacy-focused browser, Brave is expanding into AI-driven search with its Brave Search API, which offers ad-free, unbiased results.
- Exa: A startup focused on semantic search, Exa provides APIs for retrieving structured data from the web, similar to Keenable.
- Perplexity AI: While not a direct competitor, Perplexity AI offers an AI-powered search engine that synthesizes information from multiple sources, highlighting the demand for AI-native search tools.
However, Keenable’s focus on building a dedicated web index for AI agents sets it apart. Unlike Brave or Exa, which retrofit existing search infrastructure for AI, Keenable is starting from scratch with a machine-first approach.
Industry Impact: What Keenable’s Rise Means for AI and Search
Keenable’s $26 million seed round, led by Accel, signals strong investor confidence in the future of AI-native search. The implications of this shift are far-reaching:
1. Accelerating AI Adoption
AI agents are becoming increasingly sophisticated, but their effectiveness is limited by the quality of the data they can access. Keenable’s index could unlock new capabilities for AI agents, such as:
- Real-time fact-checking for news and research.
- Automated content generation with accurate, up-to-date information.
- Enterprise automation (e.g., legal research, financial analysis, or customer support).
2. Disrupting Traditional Search
If Keenable succeeds, it could challenge the dominance of Google and Microsoft in the search market. While these incumbents are unlikely to disappear, their control over information could weaken as AI agents rely more on alternative indexes.
3. New Business Models for Search
Traditional search engines monetize through ads, but AI-native search opens up new revenue streams, such as:
- Subscription-based APIs for enterprises and developers.
- Pay-per-query models for high-volume AI applications.
- Partnerships with AI companies (e.g., licensing data to LLM providers).
4. Ethical and Regulatory Challenges
As AI agents become more reliant on web indexes like Keenable’s, new ethical and regulatory questions will arise:
- Data privacy: How will Keenable handle sensitive or personal data in its index?
- Bias and fairness: Will AI agents inherit biases from the web content they access?
- Copyright and licensing: How will Keenable navigate copyright issues when indexing and redistributing web content?
These challenges will require careful navigation as the industry evolves.
The Future of AI Search: What’s Next for Keenable?
Keenable’s immediate priority is scaling its engineering team, with plans to double its headcount by the end of 2026. This expansion will likely focus on:
- Improving the crawler and indexer to handle more data sources and formats.
Read the full breakdown originally published at https://ltdeveloperblogs.github.io/posts/accel-backed-keenable-is-indexing-the-web-for-ai-agents/
Top comments (0)