DEV Community: Marcus ma

How to Build an Agentic RAG Pipeline with Real-Time Web Search

Marcus ma — Wed, 26 Aug 2026 03:39:16 +0000

TL;DR

An agentic RAG pipeline treats retrieval as a tool the AI agent can call, evaluate, and call again rather than as a fixed step.
The pipeline can search an internal knowledge base first, then use real-time web search when the available evidence is missing, weak, or outdated.
Internal documents and web results should be converted into a shared evidence format before the model generates an answer.
A reliable system must preserve URLs, publication dates, document identifiers, and the claims supported by each source.
Retrieval quality, web-search precision, citation correctness, latency, cost, and stopping behaviour should all be evaluated.

A basic RAG pipeline works well until the answer is not in the knowledge base.

Imagine an enterprise copilot that can answer questions about internal product documentation. It performs semantic search against a vector database, retrieves several relevant passages, and passes them to a language model. For questions covered by the indexed documents, the system may work remarkably well.

Then a user asks about a release announced yesterday, a recently changed regulation, or how the company’s product compares with a new competitor.

The vector database cannot retrieve information it has never indexed. A conventional pipeline may return no answer, but it may also produce a confident response from incomplete or outdated context.

Adding a Web Search API helps solve the freshness problem, but it introduces another decision: when should the system trust its internal knowledge, and when should it search the open web?

An agentic RAG pipeline places that decision inside the retrieval workflow.

What Makes a RAG Pipeline Agentic?

A traditional RAG pipeline usually follows a fixed path: transform the question into a search query, retrieve the most similar passages, add those passages to the prompt, and generate an answer.

An agentic RAG pipeline allows the model to make decisions between those stages. Retrieval becomes a tool rather than a mandatory one-time operation.

The agent can determine what information the question requires, decide which source to search, inspect the evidence, reformulate the query, and retrieve again. It can also decide that the evidence is already sufficient and skip unnecessary searches.

This does not mean every stage has to be autonomous. The strongest architectures often combine deterministic controls with a limited number of model-driven decisions. Search budgets, domain restrictions, evidence schemas, and citation requirements can remain fixed even when the agent controls query planning and routing.

Microsoft’s agentic RAG architecture guidance describes a similar pattern: retrieval is exposed as a tool that an agent can invoke while reasoning across several information sources.

For a broader comparison of the two approaches, see Agentic RAG vs. Traditional RAG: How AI Agents Improve Retrieval.

The Architecture of an Agentic RAG Pipeline

A practical pipeline can begin with internal retrieval and expand to the web only when the initial evidence is inadequate.

User query
    ↓
Intent and query planner
    ↓
Internal vector search
    ↓
Evidence grader
    ├── Sufficient ─────────────→ Answer with citations
    │
    └── Missing, weak or stale
                    ↓
             Web Search API
                    ↓
       Extract, normalise and deduplicate
                    ↓
         Optional follow-up search
                    ↓
             Answer with citations

The planner first determines whether the question contains several information needs. A broad request may need to be divided into smaller queries before any retrieval begins.

The internal retriever then searches the existing knowledge base. This stage can use vector search, keyword search, or a hybrid approach. The important point is that the retrieved documents are not passed directly to the answer generator.

An evidence grader checks whether those documents can support the requested answer. It should assess coverage, freshness and direct support—not just embedding similarity.

If the evidence is sufficient, the pipeline can answer without using the web. If it is incomplete or outdated, the agent calls the Web Search API. The resulting pages are extracted, normalised, deduplicated and added to the same evidence store used for internal documents.

This design gives web search a clear role. It is neither a permanent first step nor an unstructured last resort. It is an external evidence source invoked under defined conditions.

Build the Pipeline Around Evidence, Not Tools

The most useful way to design this system is to begin with the evidence required by the final answer. Tools are simply different ways of obtaining that evidence.

Expose Internal Retrieval and Web Search as Separate Tools

The internal retrieval tool should return the document text together with information such as the document ID, collection, section and retrieval score. The web-search tool should return a title, URL, snippet, source, publication date and, when necessary, extracted page content.

Keeping the tools separate makes routing visible. It becomes possible to determine whether the agent searched the web unnecessarily, ignored a useful internal document, or relied on a snippet when it should have inspected the complete page.

The tools should retrieve information rather than generate final answers. If a search tool silently summarises its results, the pipeline may lose the connection between the original evidence and the claims produced later.

LangGraph’s custom RAG agent guide demonstrates this separation through dedicated retrieval, document-grading and query-rewriting stages. The same principle applies even if the pipeline uses another framework or a custom orchestrator.

Add a Router and an Evidence Grader

The router decides where the search should begin. A question about an internal policy probably belongs in the knowledge base. A question containing phrases such as “latest”, “today” or “current price” is more likely to require real-time search.

Some questions need both sources. A support agent might use internal documentation to explain how a product works, then search the web for a newly disclosed vulnerability affecting one of its dependencies.

After internal retrieval, the evidence grader determines whether the results are sufficient. It should distinguish between several failure modes: no documents were retrieved, the documents are off-topic, the information is relevant but incomplete, or the information is too old for the question.

This is where corrective RAG and adaptive RAG ideas become useful. The Corrective RAG paper proposes assessing retrieved documents and using web search to extend the available information when the original corpus produces weak results.

The grader does not need to be an unrestricted agent. It can use a structured output such as sufficient, partial, irrelevant or stale, followed by a short explanation. The workflow can then route each result through predefined edges.

Normalise Internal and Web Evidence

A vector database and a Web Search API return different types of data. If those formats are passed directly to the model, it becomes difficult to compare sources, remove duplicates or produce reliable citations.

Both sources should be converted into a shared evidence structure:

{
  "source_type": "internal | web",
  "title": "Source title",
  "url_or_document_id": "Source identifier",
  "published_at": "Publication or update date",
  "content": "Relevant source passage",
  "relevance_score": 0.86,
  "supported_claims": [
    "Claim supported by this source"
  ]
}

The exact fields can change, but every evidence item needs a stable identity and a traceable origin.

Web results also require additional processing. Several pages may repeat the same press release, quote the same research paper, or reproduce an announcement without adding independent evidence. Deduplication should therefore consider the underlying source, not only whether the URLs are different.

Search-result snippets may be enough to decide which pages deserve inspection, but they are usually too limited to support important claims. When a claim matters, the pipeline should retrieve the page and preserve the relevant passage.

For a deeper explanation of web evidence and source handling, see Agentic Search: How AI Agents Search, Evaluate, and Cite the Web.

Define Search and Stopping Rules

Once an agent can search repeatedly, it needs rules for when another search is justified.

A follow-up search may be appropriate when a key subquestion has no supporting evidence, two credible sources disagree, the retrieved pages are outdated, or the current results introduce a new term that requires investigation.

The pipeline should also know when to stop. It may finish when every important claim has at least one suitable source, the requested topics have been covered, and another query is unlikely to change the conclusion.

Hard limits remain necessary. A production system should cap the number of searches, inspected pages, tokens, elapsed time or API spend. These limits protect the application when the model keeps reformulating queries without finding anything useful.

NVIDIA’s Agentic RAG Blueprint uses planning, task execution, synthesis and optional verification, while also acknowledging that the agentic path requires additional model calls and latency. That trade-off should be explicit in any implementation.

How to Evaluate the Pipeline

The final answer is only one part of an agentic RAG evaluation. The path used to produce it also matters.

Retrieval relevance measures whether the internal search found the correct documents. Fallback precision measures whether web search was called only when it added value. If the system searches the web for every question, the router is not doing useful work.

Groundedness examines whether the answer is supported by the collected evidence, while citation correctness checks whether each cited source supports the particular claim attached to it. Source quality should account for authority, freshness and independence.

Operational metrics are equally important. Web search, page extraction and repeated tool calls add latency and cost. Agent observability should therefore capture queries, routing decisions, retrieved sources, grader outputs, retries and stopping reasons.

An evaluation set should contain simple internal questions, current questions that require the web, questions that need both sources, and questions for which no reliable answer exists. The last category tests whether the system can stop and acknowledge uncertainty instead of continuing to search indefinitely.

When a Basic RAG Pipeline Is Still Better

Agentic RAG should not be the default architecture for every retrieval task.

If the knowledge base is stable, the questions are predictable and one retrieval step usually finds the necessary context, a basic RAG pipeline will be faster, cheaper and easier to evaluate.

Agentic retrieval becomes valuable when the system must choose among sources, handle multi-step questions, recover from weak retrieval or access current information. The additional complexity should solve an identifiable retrieval problem rather than merely make the architecture appear more advanced.

Conclusion

A Web Search API gives a RAG agent access to current information. The agentic pipeline decides when that information is needed, how it should be combined with internal knowledge, which sources deserve to be trusted and when the research should end.

The most reliable systems do not begin with autonomous tools. They begin with a clear evidence model, controlled routing and measurable stopping conditions.

When those foundations are in place, real-time web search becomes more than a fallback. It becomes a traceable evidence layer for answers that an internal knowledge base could not produce alone.

Agentic Search for Developers: How AI Agents Search, Evaluate, and Cite the Web

Marcus ma — Tue, 25 Aug 2026 02:52:52 +0000

TL;DR

Agentic search lets an AI agent plan queries, inspect results, identify missing evidence, and search again before answering.
It differs from traditional RAG because it can work with live web information instead of relying only on a pre-indexed knowledge base.
Search results should be treated as evidence, not as finished answers.
A production search agent needs persistent state, source-quality checks, citations, and explicit search limits.
The hardest part is often not calling the search API—it is deciding when the agent has enough evidence to stop.

Adding web search to an AI agent looks simple at first.

Define a search tool, send the user’s question to an API, return the results to the model, and ask it to write an answer. That is enough for a demo, but it is rarely enough for a dependable research system.

The first query may be too broad. The results may be outdated, duplicated, or promotional. Important evidence may be buried several pages deep, while two credible sources may disagree about the same claim.

A useful search agent must therefore do more than retrieve links. It must decide what to search for, evaluate what it finds, recognize what is still missing, and determine when further searching is no longer useful.

That is the basic idea behind agentic search.

What Is Agentic Search?

Agentic search is an iterative search process controlled by an AI agent.

Instead of sending one query and immediately generating an answer, the agent treats the request as a research task. It can break the task into smaller questions, create multiple queries, inspect individual pages, compare sources, and refine its search plan as new information appears.

Consider this request:

Which search infrastructure would work best for a customer-support agent that needs current product documentation, regional sources, and verifiable citations?

A basic search integration might submit the whole sentence as one query and summarize the first few results.

An agentic system would approach the question differently. It might first identify several decisions that need to be made:

Which services provide sufficiently fresh web results?
Which ones support geographic or domain filtering?
Do they return source URLs and publication dates?
Can they retrieve the full page when a snippet is not enough?
What are their latency and pricing characteristics?

The searches performed later in the process depend on what the agent discovers earlier. If a provider supports regional search but does not clearly document its citation metadata, the agent can create a follow-up query specifically for that gap.

This is what makes the search process agentic: the route is not completely predetermined.

Anthropic makes a similar distinction in its guide to building effective agents. Workflows follow predefined paths, while agents dynamically decide how to use tools and direct the process.

Agentic search applies that decision-making ability to information retrieval.

The Agentic Search Loop

Most agentic search systems can be understood as a feedback loop:

Understand the goal → plan the research → search → evaluate the evidence → refine or answer

The agent begins by interpreting the request. This matters because a user’s prompt does not always contain a good search query.

For example, “compare the leading AI agent frameworks” leaves several questions unanswered. What qualifies as leading? Should the comparison focus on adoption, orchestration features, deployment, observability, or enterprise support? Does the answer require current release information?

After identifying the real information needs, the agent generates one or more focused queries. It sends them to a search API, receives the results, and evaluates whether those results provide enough evidence.

If the evidence is weak or incomplete, the agent searches again.

A simplified control loop might look like this:

state = create_research_state(user_request)

while not should_stop(state):
    query = plan_next_query(state)
    results = search_web(query)
    evidence = evaluate_results(results)
    state.add(query, evidence)

answer = generate_answer(
    request=user_request,
    evidence=state.accepted_evidence,
    include_citations=True,
)

The code is not the difficult part. The real engineering decisions are hidden inside plan_next_query, evaluate_results, and should_stop.

Those functions determine whether the system behaves like a research agent or merely a language model repeatedly calling a search endpoint.

Mistral’s Agentic Search documentation describes a similar orchestration layer in which the model can search, inspect results, navigate sources, and search again as new information needs appear.

Agentic Search vs. Traditional RAG

Agentic search and retrieval-augmented generation solve related problems, but they are not the same architecture.

Traditional RAG normally begins with a prepared knowledge base. Documents are collected, divided into chunks, converted into embeddings, and stored in a vector database. When a user submits a question, the system retrieves relevant chunks and places them in the model’s context.

This works well when the information is stable and the organization controls the documents. Internal policies, product manuals, support articles, and private company data are good RAG use cases.

Agentic search is better suited to information that changes frequently, lives on the open web, or cannot be indexed in advance. It can change its query strategy during execution and investigate unexpected information discovered along the way.

Dimension	Traditional RAG	Agentic Search
Information source	Pre-indexed knowledge base	Live web or external sources
Retrieval behavior	Usually one retrieval stage	Adaptive, multi-round search
Query strategy	Based mainly on the original prompt	Changes as evidence gaps appear
Best suited for	Stable internal knowledge	Current or open-ended research
Main risk	Missing indexed information	Search loops, weak sources, and higher cost

In practice, developers do not always need to choose one or the other.

A production agent might search an internal knowledge base first. If the internal material is insufficient or the request depends on recent information, the agent can search the web and compare the new evidence with the internal documents.

RAG supplies controlled organizational knowledge. Agentic search supplies freshness and external coverage.

Building Agentic Search with a Web Search API

A Web Search API provides access to information. The surrounding agent workflow determines whether that information becomes a reliable answer.

The architecture usually contains a planner, a search tool, an evidence store, an evaluator, and an answer generator. Depending on the application, there may also be a page-content extractor, reranker, citation validator, or human-review stage.

Give the Search Tool a Clear Contract

The search tool should accept predictable, structured inputs. These may include the query, language, region, date range, allowed domains, blocked domains, and maximum number of results.

The output should also be consistent. Each result should ideally contain a title, URL, snippet, source name, and publication date. If the tool retrieves full-page content, that content must remain connected to its original URL.

A clear contract makes tool calls easier to test and inspect. It also reduces the risk that the agent will confuse a search snippet with a verified claim.

The search function should retrieve evidence. It should not silently generate the final answer.

Treat Search Results as Evidence

A high-ranking result is not automatically a trustworthy source.

Search rankings measure relevance using many signals, but they do not guarantee accuracy. A result may be outdated, promotional, copied from another page, or based on a source that is no longer available.

Before using a result, the agent should consider questions such as:

Does the page directly support the claim?
Is the publication date relevant?
Is this a primary source or a summary of another source?
Are several results repeating the same underlying report?
Does another credible source contradict it?

For high-impact claims, the agent may need to inspect the full page or confirm the information with another independent source.

The evidence should also remain connected to its citation. If the workflow summarizes pages and discards their URLs, the final model may produce an answer that sounds well researched but cannot show where its claims came from.

Preserve Research State

A multi-step search agent needs memory.

At a minimum, it should retain the queries already attempted, sources already inspected, claims supported by each source, unresolved questions, and the reason another search is needed.

Without this state, an agent may repeat the same query, inspect the same source several times, or lose the relationship between a claim and its evidence.

Graph-based orchestration works well for this type of workflow because search naturally contains branches and loops. LangGraph’s Graph API, for example, uses shared state, nodes, and conditional edges.

A search node can retrieve results. An evaluation node can determine whether the evidence is sufficient. A conditional edge can then send the workflow either back to query planning or forward to answer generation.

The framework itself is optional. The important part is that every stage leaves behind enough structured information for the next stage to make a better decision.

Decide When to Stop

Stopping is one of the most important—and easiest to overlook—parts of agentic search.

If the agent stops too early, the answer may be incomplete. If it keeps searching, latency and API costs continue to rise even when the additional results add little value.

A reliable workflow usually combines evidence-based stopping conditions with hard limits.

The agent may stop when all required subquestions have supporting evidence, important claims have citations, and the latest searches are no longer producing new information. At the same time, the system should impose a maximum number of searches, page inspections, tokens, or seconds.

The final decision should not rely entirely on the model saying, “I am confident now.”

Model confidence can be useful, but it is not a safety boundary.

Evaluating an Agentic Search System

Evaluating only the final answer is not enough.

Two agents can produce similar responses while using very different processes. One might rely on current primary sources and stop after four focused searches. Another might perform fifteen repetitive searches, use weak sources, and attach citations that do not support its claims.

A useful evaluation should examine both the result and the research trajectory.

Groundedness asks whether the answer is supported by the collected evidence. Citation correctness checks whether each linked source supports the sentence where it appears. Source quality examines authority, freshness, independence, and relevance.

Coverage measures whether the research addressed the important parts of the request. Search efficiency considers the number of queries, page inspections, tokens, API calls, and seconds required to complete the task.

The trajectory itself can also be tested. Did the agent reformulate a query after poor results? Did it recognize conflicting evidence? Did it apply date or domain filters when required? Did it stop for a defensible reason?

Anthropic’s guide to evaluating AI agents recommends evaluating both final outputs and the tool-use process that produced them. This is especially important for search agents because an apparently good answer can hide a fragile research process.

Where Agentic Search Is Most Useful

Agentic search is valuable when an answer depends on current, external, or difficult-to-predict information.

It fits research assistants, monitoring agents, fact-checking systems, competitive intelligence tools, shopping assistants, technical support agents, and products that must answer with verifiable sources.

It is less useful when the answer already exists in a stable, controlled knowledge base. In those cases, traditional retrieval may be faster, cheaper, and easier to evaluate.

Agentic behavior should be introduced because the task requires adaptive research—not simply because an agent loop is technically possible.

Conclusion

Agentic search changes retrieval from a single lookup into a managed research process.

The agent decides what it needs to learn, creates queries, evaluates sources, preserves evidence, identifies gaps, and determines when the research is complete.

That flexibility helps AI systems answer current and open-ended questions, but it creates new engineering responsibilities. Developers must control search loops, verify citations, measure source quality, preserve state, and manage cost and latency.

Calling a Web Search API is the easy part.

Building an agent that knows what to search for, what to trust, and when to stop is where the real work begins.

If you are building an agent that searches the web, what has been the hardest part to control: query planning, source quality, citations, or stopping conditions?

I would be interested to hear how you are approaching it.

AI Slop Is Becoming a Search Infrastructure Problem

Marcus ma — Mon, 24 Aug 2026 08:15:39 +0000

LinkedIn recently added a “Seems like AI slop” option to the menu attached to each post. According to the company’s chief product officer, users selected it more than one million times during its first two weeks.

The number represents reports rather than verified AI-generated posts or unique users. Even so, one million clicks is a strong signal. People are finding enough repetitive, low-value content in their feeds that they actively want a way to filter it out.

For most users, this looks like a social media moderation problem. For developers building search engines, RAG applications, research assistants, and autonomous agents, it exposes a deeper failure mode.

The web can contain millions of pages without containing millions of independent facts.

AI-Generated and Low-Quality Are Different Labels

“AI slop” has no stable technical definition.

The term can describe automatically generated spam, inaccurate summaries, repetitive LinkedIn posts, mass-produced SEO pages, or any writing that sounds recognizably machine-generated.

These categories often get grouped together, even though they represent different problems.

Authorship asks how the content was created. Accuracy asks whether its claims are true. Originality asks whether it contributes new information. Quality asks whether it helps the reader accomplish something.

An AI-content detector usually addresses only the first question.

This distinction matters for developers because authorship is an unreliable proxy for usefulness. A human can manually publish an empty article built from familiar talking points. An AI-assisted article can include original benchmarks, customer interviews, real implementation details, and carefully verified sources.

A system that treats “likely AI-generated” as equivalent to “low quality” will make predictable mistakes.

Generated status should be treated as metadata. It should not become the quality score itself.

LinkedIn Is Building a Human Feedback Dataset

LinkedIn’s announcement described AI slop as a priority and outlined new classifiers for identifying low-quality and automated content.

The reporting button adds another component: human-labeled feedback.

That feedback is valuable because people notice qualities that automated classifiers struggle to measure. An experienced developer may immediately recognize that a technical post contains no working details. A hiring manager may see that a leadership story is built entirely from recycled advice. A researcher may notice that an article contains statistics without identifiable sources.

Each click gives LinkedIn a signal that a post produced a negative quality judgment.

The signal also contains noise.

Readers have different standards for what counts as AI slop. Some react to formatting, tone, or vocabulary. Others use the label for any content they dislike. Posts written by non-native English speakers may be polished with writing tools and then mistaken for automated content.

A reporting option can also be abused by competitors, critics, or coordinated groups.

The button is useful because it collects experience at scale. Its reliability depends on how LinkedIn combines that data with other signals.

For search and recommendation developers, this is a familiar lesson: user feedback is informative, contextual, and imperfect.

Watermarks Solve a Smaller Problem

Anthropic is approaching AI-content transparency from the generation side.

Claude models launched on or after August 2, 2026 include machine-readable markings in generated text. Files such as images and documents may also include signed provenance metadata. Anthropic explains the approach in its documentation on how Claude marks AI-generated content.

A text watermark generally works by influencing token selection. The output reads normally to a person, while a detector can examine statistical patterns in the text.

This can support provenance, regulatory compliance, abuse investigation, and coordinated campaign detection. It helps answer whether a piece of text probably came from a particular generation system.

It says very little about whether the content is accurate.

A watermarked security explanation could be carefully researched and technically correct. A manually written article could contain fabricated benchmarks and invented sources.

Watermarks also become less reliable as content moves through editing pipelines. Text may be shortened, translated, paraphrased, or passed through another model before publication.

A recent empirical evaluation of AI watermarking found that paraphrasing substantially weakened several watermarking approaches and produced uncertain or incorrect classifications under some experimental conditions.

Watermarking remains useful for provenance. Search quality requires a wider set of evidence.

The Real Failure Mode Is Synthetic Consensus

Consider a typical web-enabled RAG pipeline:

A user submits a question. The application retrieves search results, extracts page content, splits it into chunks, ranks those chunks, and sends the highest-ranked material to a language model.

Now imagine that one incorrect claim is published on a small website.

Several automated news aggregators summarize it. SEO sites rewrite those summaries. Social media accounts turn the claim into short posts. More websites generate articles based on those posts.

A search query may return twenty pages that repeat the same claim with slightly different wording.

A basic retrieval system sees twenty relevant documents. The model sees apparent agreement across several sources. The user receives a confident answer.

The system has mistaken repetition for confirmation.

This is synthetic consensus: one claim is transformed into many pages, and content volume creates the appearance of independent evidence.

Keyword-based duplicate detection catches exact copies. AI-generated rewrites are more difficult because the wording changes while the underlying information remains the same.

Embedding similarity can help identify near-duplicates, although document-level similarity alone may miss pages that share only one repeated claim. Stronger systems will need to compare sources, citations, entities, and individual factual statements.

Top-K Retrieval Can Amplify the Problem

Many RAG systems optimize retrieval around semantic relevance.

Given a query, they select the chunks that appear most closely related to the user’s question. This works well when the source collection contains diverse and reliable material.

On the open web, the most semantically similar results may all be derived from the same origin.

If the top five results repeat one announcement, the model effectively receives one source five times. The repeated language increases confidence without increasing evidence.

More documents do not automatically create better context. Independence matters as much as relevance.

A retrieval pipeline should therefore consider whether its selected sources represent separate information paths.

Two articles quoting the same press release belong to one evidence cluster. A vendor announcement, an independent benchmark, a customer report, and a public dataset provide four different forms of evidence.

This distinction becomes especially important for research agents. An agent may be instructed to compare sources, yet it cannot perform a meaningful comparison when every result originates from the same claim.

What Trust-Aware Retrieval Needs to Measure

A better search and retrieval layer needs more than a binary AI-content label.

Provenance

The system should identify where a claim first appeared.

A product announcement from the company, a report quoting that announcement, and a generated summary of the report should have a visible relationship. Search results should help the model locate the primary source.

Source independence

Several URLs may still represent one source.

Retrieval systems should cluster pages that share the same citations, quotes, data, or factual structure. The final context should contain evidence from multiple independent clusters rather than several rewrites from one cluster.

Freshness

Technical information expires quickly.

API behavior, pricing, security advisories, laws, product availability, and model specifications can change within days. A polished article may rank well long after its details have become obsolete.

Published dates, update dates, and the timing of cited sources should influence retrieval.

Citation support

A page containing links is not necessarily well sourced.

The retrieval layer should check whether a citation actually supports the surrounding claim. This requires moving beyond URL counting toward claim-to-source relationships.

Duplication

Exact duplicate detection is no longer enough.

Systems need semantic deduplication at the document and claim levels. They should identify articles that preserve the same facts and reasoning while changing the presentation.

Domain-specific authority

A website’s general popularity does not guarantee expertise in every subject.

A small project repository may be the strongest source for a software change. A government regulator may be the best source for a new rule. An independent security researcher may have better evidence about a vulnerability than a large technology publication.

Authority should be evaluated in relation to the task.

Search APIs Need to Return More Than Text

A basic search response may contain a title, URL, snippet, and page content.

Agent-oriented search benefits from richer metadata: publication time, source type, citations, language, content format, and relationships between results.

This metadata helps the agent distinguish a primary source from a summary, compare dates, and avoid treating duplicate pages as independent evidence.

Search APIs designed for AI agents, including Cloudsway Search, are moving toward structured web data because raw text alone provides too little context for reliable decisions.

The retrieval layer should help an agent answer two separate questions:

What does this page say?

Why should this page influence the answer?

Most current RAG pipelines are much better at the first question.

AI Agents Make Search Quality More Important

A human browsing search results has several informal defenses.

They can recognize a suspicious domain, notice repetitive language, open multiple tabs, inspect the author, or decide that a page feels empty.

An AI agent may read and process hundreds of pages without experiencing that kind of fatigue or skepticism. It can absorb low-quality information at machine speed.

The consequences also extend beyond generating a weak summary.

A coding agent may follow outdated documentation. A shopping agent may recommend a product based on automated comparison pages. A research agent may cite several articles that all copied the same source. A compliance agent may interpret an old regulation as current.

When an agent can act on retrieved information, search quality becomes part of the application’s safety model.

Model intelligence cannot compensate for missing or misleading evidence. The retrieval layer decides what information reaches the model in the first place.

Better Content Still Has a Future

The growth of AI-generated publishing does not make content irrelevant. It changes which content remains valuable.

First-hand information becomes more important. Benchmarks, experiments, interviews, implementation failures, screenshots, datasets, and detailed case studies add information that cannot be recovered by summarizing existing search results.

Clear sourcing also becomes a competitive advantage. Articles that show where claims came from are easier for readers to verify and easier for agents to cite.

Independent judgment matters as well.

The web already has enough summaries. Useful writing explains why an event matters, which assumptions deserve scrutiny, and what changes for the reader.

AI can support research, organization, editing, and translation. The final article still needs to contribute evidence, experience, or analysis that was previously missing.

Search Will Compete on Evidence Density

LinkedIn’s million AI slop reports show that users already feel the cost of synthetic content.

Platforms are responding with report buttons, classifiers, watermarks, and provenance metadata. Each tool contributes a useful signal, and each has clear limitations.

For developers building search and RAG systems, the larger opportunity lies in evidence-aware retrieval.

The next generation of search infrastructure will need to trace claims to their origins, identify duplicate information, preserve publication context, evaluate citations, and select genuinely independent sources.

The goal is no longer to retrieve the largest number of relevant pages.

The goal is to retrieve the smallest set of sources that provides the strongest evidence.

As the web becomes easier to generate, evidence density will become one of the most valuable search signals.

DeepSeek Harness Explained: Why “Everything Is a Plugin” Matters

Marcus ma — Fri, 21 Aug 2026 02:47:07 +0000

DeepSeek Harness has already passed 170,000 stars on GitHub, and a predictable label has followed it around: Claude Code killer.

I understand the comparison. It is open source, MIT-licensed, supports multiple model providers, and can run coding-agent workflows. If you only look at the surface, it seems like another entry in the increasingly crowded list of terminal coding agents.

But that is not the interesting part.

DeepSeek Harness, or dsh, is really an attempt to make the machinery around an AI agent replaceable. The model adapter is a plugin. The tool registry is a plugin. Sessions, sandboxes, telemetry, interfaces, and even the default agent loop are plugins.

That makes DSH less like a free copy of Claude Code and more like a kit for building the coding agent—or research agent, or internal automation agent—you actually want.

TL;DR

An agent harness is the runtime around a model: context, tools, state, permissions, execution, and the loop that connects them.
DeepSeek Harness applies one plugin system to nearly every part of that runtime.
It gives developers more architectural control than a typical finished coding agent, but also more setup and maintenance work.
Claude Code and Codex are still better fits when you want a polished product. OpenCode is a closer comparison if you want an open, provider-agnostic coding agent.
DSH is currently a developer preview, so treat it as a platform to explore rather than infrastructure you can adopt without evaluation.

First: What Is an Agent Harness?

A language model cannot edit a repository, run tests, search documentation, or ask you to approve a shell command on its own.

It needs a runtime that decides:

what context reaches the model;
which tools the model can call;
how tool arguments are validated;
where commands are executed;
how session state is stored;
when the agent should continue or stop;
which actions require human approval.

That runtime is the harness.

This is why the same model can feel surprisingly different across two coding tools. Model quality matters, but so do the system prompt, context strategy, tool schemas, permission boundaries, and recovery behavior.

The model supplies the reasoning. The harness determines how that reasoning turns into action.

What DSH Does Differently

Plenty of developer tools have plugins. DeepSeek Harness goes further by making the runtime itself composable.

DSH is built on Cordis, a framework in which plugins contribute services, typed events, and reversible effects to a shared context. According to the official architecture guide, there is no privileged core that must be patched whenever you want to change the product.

Instead, a running DSH instance is assembled as a plugin tree.

The base layer provides things such as model adapters, tools, persistence, sandboxing, approvals, credentials, and telemetry. A profile then adds the pieces needed for a particular environment. DSH currently ships web and headless profile templates.

The practical result is that you can swap one capability without rebuilding everything around it.

For example, a team could:

change the model provider while keeping its existing tools;
move shell execution from a laptop to a remote sandbox;
replace the default loop with its own orchestration strategy;
build a different UI on top of the same session system;
intercept tool calls to add approvals, logging, or organization-specific policy.

The default loop is not hidden magic. It is another registered service.

That is the part of DSH I find most interesting. It treats agent behavior as infrastructure that can be inspected and replaced, not merely configured around the edges.

A Concrete Example: Moving Execution

Imagine that your internal coding agent starts on developer laptops. Later, security requires every shell command to run in an isolated remote environment.

In many agent implementations, filesystem access, subprocess execution, terminal state, and tool definitions have grown together. Moving execution means rewriting several layers and then checking every place that assumed a local machine.

DSH defines capability seams between providers and consumers. Its filesystem and subprocess providers can share an execution environment, so replacing the backend can move Bash, PTY, and language-server operations together.

That does not make the migration automatic. You still need to implement and secure the provider. But it gives the change a defined architectural boundary, which is much better than discovering the boundary through production bugs.

DSH vs Claude Code, Codex, and OpenCode

These projects overlap, but they optimize for different jobs.

Tool	Best description	Where it fits
Claude Code	A polished coding agent centered on Claude	You want a strong terminal or IDE workflow with sensible defaults
Codex	A connected coding-agent experience across CLI, IDE, cloud, and desktop	You want local and managed workflows built around OpenAI models and services
OpenCode	An open-source, provider-agnostic coding agent	You want an open coding tool you can configure and use directly
DeepSeek Harness	A composable runtime for assembling agents	You want to replace or own parts of the agent architecture itself

Calling DSH a Claude Code alternative is not wrong, but it hides the trade-off.

Claude Code gives you an integrated product. DSH gives you architectural control. The first removes decisions; the second exposes them.

The same applies to Codex. Codex CLI is open source, so this is not a simple “open versus closed” comparison. The difference is that DSH makes more of the surrounding runtime—including the loop—part of one replaceable composition model.

OpenCode may be the closest practical comparison, but even there the emphasis differs. OpenCode is primarily a flexible coding agent. DSH is useful when you want to reshape or embed the agent itself.

Who Should Actually Try It?

DSH makes sense if you are:

building an internal agent platform;
experimenting with several model providers;
designing custom tools, permissions, or approval flows;
moving agent execution into your own sandbox;
testing alternative agent loops;
embedding an agent into another application.

If you mainly want help writing and reviewing code every day, DSH may be more infrastructure than you need. Claude Code, Codex, or OpenCode will get you to a productive workflow faster.

A useful rule of thumb is this:

Choose a coding agent when you want to delegate coding. Choose an agent harness when you want to control how delegation works.

What It Still Costs

MIT-licensed does not mean free to operate.

You still need models, compute, storage, sandboxes, monitoring, and people who understand the system well enough to maintain it. If the agent works with current external information, you also need a reliable way to search and read the web without losing source metadata.

This is where the modular approach becomes useful. Search, page reading, execution, and orchestration do not have to be bundled into one agent product.

For example, an agent can use a search service to retrieve current, source-backed information, a reader to turn pages or documents into model-ready content, and a sandbox to isolate code execution. Those capabilities can evolve separately as long as their interfaces remain stable.

At Cloudsway, this is the layer we work on: Search and Reader provide live web information and structured content, while Scalebox provides an isolated execution environment. DSH offers an interesting foundation for wiring capabilities like these into a custom agent without making them permanent parts of one monolithic loop.

The Developer Preview Label Matters

The repository is explicit about its current status: DeepSeek Harness is a developer preview, and compatibility-breaking changes should be expected.

That should affect how you evaluate it.

Before using DSH in a real workflow, I would want:

pinned versions and repeatable integration tests;
a clear review process for plugins;
strict separation between untrusted content and powerful tools;
isolated command execution;
approval policies for destructive or sensitive actions;
logs for model requests, tool calls, and permission decisions;
an upgrade plan for configuration and plugin changes.

The security point is especially important. An agent may read repositories, web pages, documentation, issues, or reusable instructions that contain content written by someone else. If the same agent can execute commands or access credentials, prompt injection becomes an execution-boundary problem—not just a prompting problem.

Modularity gives you places to enforce controls. It does not enforce them for you.

Trying DSH Locally

With Node.js installed, the quickest way to launch the Web profile is:

npx @deepseek-ai/dsh web

By default, DSH starts its local Web UI at http://127.0.0.1:3080.

If you want to inspect the source instead:

git clone https://github.com/deepseek-ai/deepseek-harness.git
cd deepseek-harness
pnpm install
pnpm run build
pnpm dsh web

The repository is moving quickly, so check the official README before relying on setup instructions copied elsewhere.

For a longer walkthrough of profiles, bundles, providers, and the Web UI, we also published a DeepSeek Harness architecture and quick-start tutorial.

Final Take

DeepSeek Harness probably will not make paid coding agents obsolete. That is the wrong benchmark for the project.

Its more interesting contribution is making the harness layer visible. It encourages developers to treat model access, context, tools, state, permissions, execution, and orchestration as separate design decisions.

For most people, a finished coding agent is still the practical choice. For teams building their own agent infrastructure, DSH is worth studying—even if they never use it in production.

The question is not whether DSH can imitate Claude Code. It is whether developers want enough control to build a different kind of agent.

If you have tried DSH, which part of the runtime would you replace first: the model, the tool layer, the sandbox, or the agent loop?

Cursor Origin: Why AI Coding Is Moving Beyond the Editor

Marcus ma — Wed, 19 Aug 2026 03:28:28 +0000

On August 17, Cursor opened the early beta of Origin, a new code-hosting service for paid users. It includes repositories, pull requests, code browsing, GitHub synchronization, and integrations with services such as Vercel, Depot, and Buildkite.

The launch quickly reached Hacker News and X, where much of the discussion framed Origin as a possible GitHub alternative. That comparison makes an effective headline, although the more interesting story lies in the direction of travel. AI coding companies are expanding beyond code generation and moving into the infrastructure where software work is stored, reviewed, tested, and shipped.

Origin offers an early view of what an agent-native development platform might look like.

The Editor Was the Easiest Place to Start

The first wave of AI coding tools lived inside the editor. Autocomplete suggested the next line, chat interfaces explained unfamiliar functions, and code-generation tools turned short instructions into working snippets.

That model worked because the developer remained responsible for the wider process. A person selected the relevant files, checked the output, ran the tests, created the branch, opened the pull request, and followed the change through review.

Coding agents have widened that scope. They can explore repositories, modify several files, run commands, investigate failures, and prepare pull requests. Cloud-based agents can continue working after the developer closes the editor. Some can also monitor CI results or respond to review comments.

Once an agent starts handling work that lasts longer than a single coding session, the editor becomes only one surface in a much larger system.

AI Coding Agents Need More Than Code Access

A useful coding agent needs access to the source code, but source code represents only part of the development context.

Consider a request such as “fix the checkout failure introduced in the latest release.” The agent may need to inspect the issue, identify the relevant service, review recent commits, understand repository conventions, reproduce the failure, run tests, create a branch, open a pull request, and respond to comments from reviewers.

Each step produces new context. Test results influence the next edit. Review comments reveal requirements that may not appear in the original task. CI failures provide information about environments the agent cannot reproduce locally. Deployment results show whether the change worked outside the development machine.

For a human developer, these signals are spread across familiar tools. An autonomous agent needs a structured way to read them and act on them.

This changes the role of the repository. It begins to serve as the shared operating record for developers, agents, reviewers, and deployment systems.

The Repository Is Becoming an Agent Workspace

Traditional code hosting platforms are organized around human collaboration. Developers create branches, submit pull requests, leave comments, approve changes, and trigger automation.

Agent-driven development adds another participant with different requirements. An agent may need an isolated environment, explicit permissions, persistent task state, machine-readable feedback, and a clear history of every action it has taken.

The repository therefore becomes much more than a destination for generated code. It provides the boundaries within which an agent works.

This helps explain the significance of Cursor Origin. Its current feature set covers familiar code-hosting functions, while the surrounding product direction points toward tighter integration with agents. Cursor has already connected its coding experience with cloud agents, pull-request review, remote environments, and multi-repository workflows. Hosting the repository brings those pieces closer together.

The immediate product may look familiar. The architecture around it is designed for software work that increasingly involves autonomous systems.

Why AI Coding Companies Want the Whole Workflow

Coding agents improve when they can observe the results of their own work.

An agent that only generates a patch receives a limited feedback loop. It may never see whether the patch passed CI, satisfied the reviewer, or survived deployment. An agent connected to the full workflow can use those outcomes to decide what to do next.

Owning more of the development loop offers several advantages. Context can follow a task from the initial request through implementation and review. Environments can be prepared consistently. Permissions can be enforced at each step. Agent actions can be logged, evaluated, and resumed later.

It also gives the platform a clearer view of how software is produced. The useful data extends beyond prompts and generated code. It includes accepted changes, rejected suggestions, recurring failures, review patterns, and deployment outcomes.

This is why repository hosting matters strategically. It connects the AI coding interface to the systems that determine whether generated work is actually useful.

Origin’s integrations with Vercel, Depot, and Buildkite reinforce this direction. They connect repositories with deployment, builds, and CI infrastructure, allowing the workflow to continue after the code has been written.

GitHub Remains Difficult to Replace

Git repositories are portable. Development workflows are much less portable.

GitHub’s position comes from years of accumulated developer identities, project histories, integrations, organizational policies, automation, and community activity. For enterprises, audit logs, access controls, security processes, and existing vendor relationships can matter as much as the repository itself.

Open-source projects also depend on GitHub’s network effects. Issues, pull requests, contributor profiles, stars, forks, discussions, and Actions form a public collaboration layer that would be expensive to recreate elsewhere.

Origin’s GitHub synchronization points toward a gradual transition. Teams can experiment with a new agent-oriented environment while maintaining their existing GitHub workflow. Comments and reviews can continue to move between the two systems instead of forcing an immediate migration.

This hybrid approach is likely to shape the next phase of the market. AI coding platforms can build new workflow layers around existing repositories before asking organizations to move their source of record.

The Next Platform Lock-In May Live Above Git

Code portability will remain important, but a new kind of platform dependency is beginning to form around agent context.

An agent may accumulate task histories, repository instructions, environment configurations, tool permissions, reusable skills, memories, and feedback from previous runs. These elements influence how reliably it works, yet they do not currently share a universal portability standard.

Moving a Git repository is straightforward. Moving the complete working context of an agent may be considerably harder.

This creates an important question for development teams: who owns the operational memory generated by coding agents?

The answer will affect more than vendor choice. Teams will need to consider whether agent histories can be exported, whether permissions are understandable, whether automated actions are auditable, and whether workflows can continue if the underlying agent platform changes.

Repository hosting gives AI coding companies a strong foundation for this context layer. It also raises the stakes for interoperability and governance.

Agent-Native Development Extends Beyond the Repository

Even a well-connected repository cannot contain every piece of information an agent needs.

Software development constantly depends on external context: updated documentation, newly disclosed vulnerabilities, package releases, service incidents, API changes, and technical discussions. An agent working with outdated information can produce a plausible solution that has already become obsolete.

Real-time information access will therefore become another important part of the agent stack. Services such as Cloudsway Search provide structured web data for AI agents, allowing them to retrieve current information and support decisions with external sources.

This fits into the same broader shift. AI coding products are evolving into systems that coordinate models, repositories, execution environments, development tools, and live information. Code generation remains one component of that system.

What Origin Tells Us About the Future of AI Coding

The next stage of AI coding will probably be defined by workflow ownership.

Model quality will continue to matter, yet many leading products already provide access to several models. The larger differences will come from how effectively each platform preserves context, manages permissions, connects tools, verifies work, and helps agents recover from failure.

Repositories sit at the center of those capabilities. They contain the code, record proposed changes, connect to testing systems, and provide the review process through which software becomes trusted.

Cursor Origin has arrived early in this transition. Its first version may appeal mainly to teams already using Cursor, and GitHub will continue to hold a powerful position across open-source and enterprise development. The broader direction is still clear: AI coding companies want to participate in a much larger share of the software lifecycle.

The AI coding war began inside the editor. It is now spreading across the entire development stack.

Building an AI News Aggregator Without Drowning in Duplicate Stories

Marcus ma — Wed, 19 Aug 2026 02:46:54 +0000

Most AI news aggregator demos stop at the pleasant part: run a search, hand the results to an LLM, and render a neat summary.

The frustrating part begins on the second refresh.

An article from last month appears next to today's announcement. Three different URLs turn out to be the same press release. Five publishers repeat one wire story, making it look as if five independent sources confirmed the news. Then the summary introduces a detail that was not in any of the retrieved pages.

At that point, the project is no longer a search box with a summarizer attached. It is a small editorial system, and most of the work sits between retrieval and generation.

This post walks through that middle layer. The example is a feed that tracks AI chip export controls, but the same approach works for product launches, regulatory updates, scientific news, or any other narrow topic that changes often.

The search query is not the product

Starting with a broad query such as AI chips creates an impossible filtering job. The phrase covers hardware releases, benchmark results, earnings calls, research papers, stock commentary, supply-chain rumors, and government policy.

For a useful feed, the topic needs boundaries. In this case, I would limit it to new export rules, enforcement actions, official statements, company responses, and meaningful supply-chain effects. I would also use a seven-day window and exclude product reviews, undated explainers, and opinion pieces that do not add new reporting.

That scope can be expressed through a handful of short searches:

AI chip export controls
advanced semiconductor export restrictions
AI accelerator export license
chip export controls company response

This works better than a single oversized query. A regulator may write about an “export licensing requirement” while a publisher calls the same change an “AI chip restriction.” Several small queries catch that variation and are easier to debug when irrelevant results get through.

The retrieval source matters too. RSS is excellent for publishers that are already on a watch list. A dedicated News API is convenient when normalized article metadata is the priority. Web search is useful when the important source may be a regulator, a company newsroom, or a specialist publication that was not known in advance.

There is no need to choose only one. A practical feed can use RSS for known sources and web search for discovery.

Search results are pages, not stories

The first pass should be cheap. Before downloading full pages or calling a model, inspect the title, URL, snippet, domain, and publication date.

A title that does not contain the expected company, regulator, country, or policy term is usually easy to reject. Category pages and tag indexes can go as well. If a result has no publication date, treat the date as unknown rather than assuming it is recent. Search freshness filters help, but pages are sometimes updated or republished in ways that make old reporting look new.

It is worth keeping the original title, URL, publisher, and date even when the full article is fetched later. Those fields become part of the audit trail behind the final summary.

Here is a small Python baseline. It runs several searches, asks for results from the last week, removes repeated URLs, and sorts what remains. The example uses the Cloudsway Smart Search API because it can return both search metadata and extracted page text; the rest of the pipeline is provider-independent.

import os
from urllib.parse import urlsplit, urlunsplit

import requests


API_URL = "https://aisearchapi.cloudsway.net/api/search/smart"
HEADERS = {"Authorization": os.environ["CLOUDSWAY_API_KEY"]}


def canonical_url(url: str) -> str:
    parts = urlsplit(url)
    return urlunsplit(
        (parts.scheme, parts.netloc.lower(), parts.path.rstrip("/"), "", "")
    )


def search_news(query: str) -> list[dict]:
    response = requests.get(
        API_URL,
        headers=HEADERS,
        params={
            "q": query,
            "count": 20,
            "freshness": "Week",
            "enableContent": "true",
            "mainText": "true",
            "contentType": "TEXT",
        },
        timeout=20,
    )
    response.raise_for_status()
    return response.json().get("webPages", {}).get("value", [])


queries = [
    "AI chip export controls",
    "advanced semiconductor export restrictions",
    "AI accelerator export license",
]

seen = set()
articles = []

for query in queries:
    for item in search_news(query):
        url = canonical_url(item["url"])
        if url in seen:
            continue

        seen.add(url)
        articles.append(
            {
                "title": item.get("name"),
                "url": url,
                "published": item.get("datePublished", ""),
                "text": item.get("mainText", ""),
                "score": item.get("score", 0),
            }
        )

articles.sort(
    key=lambda article: (article["published"], article["score"]),
    reverse=True,
)

print(articles[:10])

Set CLOUDSWAY_API_KEY in the environment before running the script. The quick start covers authentication, and the search reference lists the available parameters.

This code is intentionally incomplete. It retrieves candidates and removes exact URL duplicates. The harder duplicate problem comes next.

The duplicate problem has more than one layer

Some duplicates are mechanical. Tracking parameters, fragments, and trailing slashes create different URLs for the same page. Normalizing the URL handles many of these.

Syndication is less obvious. The same article may appear on several domains with a slightly modified headline. Comparing normalized titles, opening paragraphs, named entities, and content hashes can catch most copies.

But two articles about the same event are not necessarily duplicates. A government notice and a chipmaker's response belong to one story, yet both may be valuable. Deleting either one loses context; displaying them as separate events makes the feed repetitive.

This is where event clustering becomes more useful than another deduplication rule.

For each candidate, build a compact representation from the headline, entities, publication time, and central claim. Group candidates that share the same entities and describe the same development within a reasonable time window. Embeddings are helpful when headlines use different language, but they should not decide on their own. Two articles can sound similar while referring to different rules, countries, or dates.

Inside a cluster, choose a lead source based on relevance, recency, source quality, and completeness. Keep the other independent sources attached. An official document may be best for establishing what changed, while reporting from a specialist publication may better explain the commercial impact.

A useful sanity check is to ask where each article's information originated. Ten sites repeating one wire report are still one line of reporting.

Let the model in late

The LLM should see an event cluster, not a raw page of search results.

Give each retained source an ID such as S1, S2, and S3, then provide only the title, date, relevant passage, and URL metadata returned by retrieval. Ask the model for a neutral headline, a short summary, why the event matters, and the source IDs behind each factual statement.

The important restriction is that the model cannot create a URL. It can only refer to an ID from the supplied evidence. After generation, the application maps the IDs back to stored URLs and rejects anything outside the set.

That still does not make every summary correct. A real page may be cited for a claim it does not support. The final check should compare the claim with the cited passage, not merely confirm that the URL exists.

The output also needs a way to express uncertainty. If credible sources disagree, preserve the disagreement. If there is not enough evidence, say so. For a sensitive story, requiring a primary source or two genuinely independent reports is usually safer than letting the model turn a weak signal into a confident update.

What I would add before scheduling it

The next production feature would not be a nicer UI. It would be logging.

Store the query that found each result, the filters it passed, its cluster assignment, and the source IDs used in the summary. When a bad item appears in the feed, this makes it possible to tell whether retrieval, date handling, clustering, ranking, or generation failed.

Caching matters for the same reason. A monitoring job should not repeatedly fetch and summarize pages that have not changed. Retries need limits and backoff, and a single domain should not be allowed to dominate the feed just because it publishes aggressively.

I would also keep a manual review path for low-confidence clusters and high-impact claims. Automation is useful here because it reduces a noisy stream to a manageable set of evidence. It should not remove the ability to inspect how a conclusion was reached.

Finally, an aggregator should link to original reporting instead of reproducing full articles. Facts can be summarized, but the article's language, images, and other protected expression come with separate copyright and licensing considerations.

Closing thought

The summarizer is the most visible part of an AI news product, but it is not the part that makes the feed trustworthy.

That comes from narrower queries, careful date handling, separating copied articles from independent coverage, grouping pages into events, and keeping generated statements tied to material the system actually retrieved.

If you have built a monitoring feed, I am curious how you handle event-level duplication. Do embeddings work well enough for your topic, or have you ended up combining them with entity and date rules?

Note: AI tools were used to help edit this post.

I Tried DeepSeek Harness: Architecture and Quick Start

Marcus ma — Mon, 17 Aug 2026 10:30:15 +0000

I recently spent some time exploring DeepSeek Harness, also known as dsh. The project describes itself as an open-source agent harness, but what makes it interesting is not another chat interface or a new model. It is the layer around the model: the system that connects tools, sessions, storage, execution policies, and user interfaces into a working agent.

The idea at the center of the project is simple: everything is a plugin. That sounds like a familiar software slogan, but DeepSeek Harness applies it unusually broadly. Model adapters, the tool registry, session logs, the agent loop, storage, sandbox policies, credentials, and interfaces all participate in the same composition system.

I wanted to understand what that architecture looks like in practice, how quickly the project can be run locally, and what questions a team should ask before building on top of it.

Why the Harness Layer Matters

A capable model is only one part of an AI agent. A useful agent also needs a way to receive context, call tools, maintain state, execute actions, request approval, recover from failures, and return results that people can inspect.

That surrounding system is the harness. It explains why two agents using the same model can behave very differently. Their tool definitions, permission boundaries, context delivery, execution loops, and feedback signals may be completely different. When an agent performs poorly, changing the model is therefore not always the most useful response. The real problem may be how the rest of the system is assembled.

DeepSeek Harness makes this surrounding layer visible. Instead of locking the model, tools, memory, and interface into one fixed application, it treats them as components that can be configured and replaced. This makes the project useful not only as an agent application, but also as a public reference for thinking about agent runtime design.

The official repository currently describes DeepSeek Harness as a developer preview released under the MIT License. The preview label is important: the project is ready to explore, but compatibility-breaking changes should still be expected. For now, I would treat it as a strong prototyping environment rather than a stable production contract.

How “Everything Is a Plugin” Works

DeepSeek Harness is built on Cordis, which provides the composition model underneath the runtime. Plugins contribute services, typed events, and reversible effects to a shared context. The model adapter, agent loop, tool registry, session log, persistence layer, sandbox policy, telemetry, credentials, and interface can all be mounted through the same system.

Figure 1: A simplified view of the DeepSeek Harness plugin architecture and its shared Cordis context.

This structure creates useful boundaries. A team can replace a model adapter without rebuilding its tools, change the storage layer without rewriting the agent loop, or test a different interface while keeping the underlying runtime intact. Search, storage, approval policies, and observability can evolve independently, which matters because those parts often change at different speeds in a real agent project.

Cordis adds another useful property: plugins are designed to manage their own runtime effects. Its architecture documentation describes temporal composability, where removing a component reverses the effects it introduced, and spatial composability, where components declare dependencies and react to changes in their surrounding context. In plain language, the framework is designed for systems whose components may appear, disappear, or depend on one another while the application is running.

That model fits AI agents particularly well. Different sessions may need different tools, policies, sandboxes, model routes, or interfaces. A modular runtime makes those variations easier to express without turning every experiment into a separate application.

DeepSeek Harness assembles these components through profiles and bundles. A profile is a named composition, and the project provides web and headless profile templates. A bundle packages configuration and the code it mounts. If you want to see what a profile will load before starting it, the CLI can print the resolved configuration:

dsh --profile web --dump-config

This is especially helpful when a setup grows beyond the default configuration and you need to understand where a model, tool, or policy enters the plugin tree.

Running DeepSeek Harness Locally

The fastest way to explore the project is through its Web UI. With a supported Node.js version installed, open a terminal in a safe test workspace and run:

npx @deepseek-ai/dsh web

The interface is served locally at:

http://127.0.0.1:3080

Figure 2: DeepSeek Harness running locally through its Web UI.

Once the interface opens, configure a model credential under Settings → Models, choose a safe workspace, and start a new session. A simple first request is enough to confirm that the model can inspect the selected directory:

Summarize this workspace and identify its main packages.

The directory from which the command is invoked becomes the default filesystem location, although a new Web UI session still asks you to select a workspace before the composer becomes available.

If your goal is to inspect the implementation or develop plugins, it is better to run the project from source:

git clone https://github.com/deepseek-ai/deepseek-harness.git
cd deepseek-harness
pnpm install
pnpm run build
pnpm dsh web

The project’s development guide currently lists Node.js 22.19+ or 24+, Corepack-enabled pnpm, and Git as prerequisites. A model API key is not necessary for reading and building the repository, but it is required for running real-model examples.

Bringing Current Web Information Into an Agent

After running the basic interface, I was particularly interested in web search. Models often need information that changes after training, such as new documentation, product releases, regulations, and research. Search is therefore a natural harness capability, but it also needs to remain separate from the model itself. Teams may change search providers, apply different source policies, or add internal retrieval without wanting to rebuild the rest of the agent.

DeepSeek Harness follows that separation. The model-facing web_search tool is distinct from the provider that supplies the results. Its provider-neutral request currently includes a query and an optional result limit, while providers return normalized sources with URLs and, when available, titles, snippets, and publication dates. The official repository includes provider packages for DeepSeek, Exa, and Perplexity.

For a local proof of concept, I used Cloudsway SmartSearch as the external retrieval layer for a DeepSeek Harness research session. The test returned current titles, source URLs, dates, and summaries that the agent could use while preparing its answer.

Figure 3: Results returned during a local DeepSeek Harness test using Cloudsway SmartSearch.

This was an experimental local setup rather than an official or bundled DeepSeek Harness integration. I have deliberately left the compatibility layer out of this quick-start article because the current setup is not yet a clean one-command installation. Anyone evaluating the search API itself can review the available interfaces in the Cloudsway Search documentation.

The cleaner long-term solution would be a native search provider that calls the external API directly, registers with the Harness web runtime, and normalizes its response behind the standard web_search tool. That would preserve the central architectural idea: the agent loop should be able to use search without being tightly coupled to a particular retrieval service.

What I Would Evaluate Next

The Web UI makes the first run easy, but a successful demo is not enough to establish that an agent runtime is ready for real work. The next step should be a small set of representative tasks with clear success criteria. A coding agent might need to read a workspace, modify a file, run a command, and stop for approval at the correct moment. A research agent might need to find current sources, distinguish official documentation from third-party commentary, and preserve citations in its final answer.

During those tests, I would pay close attention to compatibility, permissions, state, failure recovery, and observability. It should be clear which tools can read files, write changes, execute commands, or access credentials. Model and search timeouts should fail in a predictable way. Session events should be inspectable, and critical tools should remain portable if the team later changes its model or runtime.

Version pinning also matters because DeepSeek Harness is still a developer preview. A prototype should record the exact configuration and package versions that produced its results. Otherwise, a fast-moving plugin API can make it difficult to reproduce an experiment several weeks later.

The most useful evaluation question is not “How many tools can this agent call?” It is whether the complete harness improves the reliability of a real workflow. More components only help when their boundaries make the system easier to test, replace, and audit.

Frequently Asked Questions

What is DeepSeek Harness?

DeepSeek Harness, or dsh, is an open-source agent runtime developed by DeepSeek AI. It assembles models, tools, sessions, sandboxes, agent loops, storage, and interfaces through a plugin-based architecture.

Is DeepSeek Harness open source?

Yes. Its official repository is available under the MIT License. The project is currently a developer preview, so users should expect its APIs and configurations to change.

How do I install DeepSeek Harness?

The quickest option is to install a supported Node.js version and run npx @deepseek-ai/dsh web. Developers who want to inspect or modify the implementation can clone the official repository and build it with pnpm.

Is DeepSeek Harness ready for production?

DeepSeek does not currently present it as a production-stable release. Teams should validate version compatibility, permissions, failure handling, credential management, state persistence, and observability before using it for production workloads.

Final Thoughts

DeepSeek Harness is interesting because it treats the agent runtime—not only the model—as a first-class engineering surface. Its Cordis-based architecture provides a consistent way to compose model access, tools, sessions, policies, sandboxes, storage, and interfaces.

For now, its strongest use case is exploration. Launching the Web UI takes only one command, but the more valuable exercise is inspecting the plugin tree and identifying which parts of an agent your own team needs to keep replaceable. Search is one example, but the same logic applies to models, permissions, storage, and execution environments.

The project is moving quickly, and production teams should plan for change. Even so, it offers a concrete and unusually open example of how the harness layer around an AI model can be designed.

Sources

Disclosure: I work with Cloudsway. I used Cloudsway SmartSearch for the local retrieval test described above. This is not an official or bundled DeepSeek Harness integration.

AI assistance: AI tools were used to help structure the article, edit the English, and create the illustrations. I reviewed the technical claims against the official documentation and local test results before publication.

Sales Prospecting with Web Search: How to Research Prospects Before Outreach

Marcus ma — Thu, 13 Aug 2026 03:07:32 +0000

Sales Prospecting with Web Search: How to Research Prospects Before Outreach

TL;DR

Sales databases are good at telling you who to contact.

Web search can help explain why now.

Useful prospecting signals include:

funding
hiring
product launches
leadership changes
partnerships
geographic expansion

A simple workflow looks like this:

Target Account
     ↓
Search Objectives
     ↓
Web Search API
     ↓
Recent Company Information
     ↓
Signal Classification
     ↓
Prospect Brief
     ↓
Personalized Outreach

The goal is not to replace your CRM or sales database.

Web search works best as the live context layer around them.

The Problem: Contact Data Is Not Enough

Suppose your sales database gives you this:

Company: Acme
Employee Count: 500
Industry: SaaS
Contact: VP of Engineering

Useful?

Yes.

Enough to write a relevant message?

Probably not.

The harder question is:

Why should I contact Acme right now?

To answer that, a rep often starts another research process:

Open company website
      ↓
Search recent news
      ↓
Check product announcements
      ↓
Look for hiring
      ↓
Search funding / partnerships
      ↓
Find an outreach angle

Doing this for five strategic accounts is manageable.

Doing it for hundreds of accounts becomes expensive very quickly.

That is where web search can become part of the prospecting workflow itself.

Step 1: Define the Signals Worth Looking For

You do not need a complete company profile before every outreach.

You need enough recent context to answer:

What changed at this company?

Could that change create a relevant need?

A few signal categories are especially useful.

Funding

Look for:

new funding rounds
acquisitions
major investment announcements

Funding alone does not prove buying intent.

But it can indicate new budgets, expansion, or upcoming infrastructure investment.

Hiring

Hiring can reveal where a company is investing.

For example:

10 new AI engineering roles

is more useful than simply knowing the company is growing.

You can go further and look for:

Infrastructure hiring
Security hiring
AI / ML hiring
International sales hiring
Developer relations hiring

The team being expanded often matters more than the raw number of jobs.

Product Launches

New products can create new technical requirements.

Useful signals include:

new product lines
beta releases
enterprise plans
API launches
new integrations
major feature releases

A launch can also tell you which direction the company is moving strategically.

Leadership Changes

A new executive may change:

priorities
tooling
vendors
budgets
operating processes

For enterprise sales, these changes can be useful triggers for deeper account research.

Partnerships and Expansion

Look for:

new markets
regional expansion
partnerships
integrations
channel agreements

Expansion often creates requirements that did not exist before.

Step 2: Turn Sales Questions Into Search Objectives

The easiest implementation is keyword-based.

For every account, you could search:

[company] latest news

[company] funding

[company] hiring

[company] product launch

[company] partnership

[company] expansion

This works, but a more useful AI workflow can express the actual research objective.

Instead of:

Acme hiring

you might ask:

Find recent evidence that Acme is expanding
its AI or infrastructure engineering teams.

Instead of:

Acme news

you could use:

Find important product, funding, partnership,
or expansion announcements from Acme this month.

For sales prospecting, the query should be designed around the signal you want to detect.

Step 3: Use Search as the Live Research Layer

The architecture can remain pretty simple:

CRM / Account List
       ↓
Company
       ↓
Generate Search Queries
       ↓
Web Search API
       ↓
Recent Web Results

At this point, your application has current information.

But raw search results still are not a prospect brief.

That requires another processing step.

Step 4: Classify the Results Into Sales Signals

Suppose we search a target SaaS company and retrieve these results:

1. Company launches a new enterprise product

2. Company opens engineering roles in Singapore

3. Company announces an APAC partnership

A downstream application or LLM can classify them:

[
  {
    "type": "product_launch",
    "importance": "high"
  },
  {
    "type": "hiring",
    "importance": "medium"
  },
  {
    "type": "expansion",
    "importance": "high"
  }
]

Now the workflow starts turning general web information into sales intelligence.

Step 5: Combine Related Signals

Individual facts often become much more useful when combined.

For example:

Signal 1:
Acme launched an enterprise product.

Signal 2:
Acme is hiring infrastructure engineers in Singapore.

Signal 3:
Acme announced an APAC partnership.

Individually, these are just company updates.

Together, they suggest:

Acme may be increasing its enterprise focus
while expanding its APAC operations.

That creates a much stronger research hypothesis.

The system could produce:

Recent signal: APAC expansion and increased enterprise focus.

Possible relevance: The company may need infrastructure capable of supporting regional growth.

Suggested research angle: Investigate whether its current stack supports multi-region deployments.

Notice the wording.

The system is not claiming:

Acme definitely needs our product.

It is generating a research hypothesis for the rep to validate.

That distinction matters.

Step 6: Build a Prospect Brief

A useful prospect brief does not need to be long.

Something like this is usually enough:

Company:
Acme

Recent Signals:
- Enterprise product launch
- APAC partnership
- Infrastructure hiring in Singapore

What May Be Changing:
Increasing enterprise and regional expansion.

Potential Relevance:
May create additional infrastructure and
multi-region requirements.

Sources:
- Company announcement
- Careers page
- Industry coverage

The rep can review the evidence before deciding whether the signal is relevant.

Web Search vs Sales Databases

Web search and sales intelligence databases solve different problems.

	Sales Database	Web Search
Main purpose	Structured account/contact data	Current company context
Typical question	Who works there?	What changed recently?
Data	Roles, emails, company size	News, launches, hiring, funding
Fresh discovery	Limited	High
Best use	Contact discovery	Trigger and context research

A sales database might tell you:

Jane Smith
VP of Engineering
Acme

Web search might tell you:

Acme launched an AI product.

Acme is hiring 10 infrastructure engineers.

Acme is expanding into Europe.

Together, they answer:

Who should I contact?

Why might this account be worth researching now?

That is why I would treat web search as a complement to CRM and enrichment systems rather than a replacement.

Example: Automating Account Research

Suppose your sales team has 100 target accounts.

A scheduled workflow could run:

Account List
    ↓
For Each Company
    ↓
Search Recent Signals
    ↓
Remove Duplicates
    ↓
Classify Results
    ↓
Rank Important Signals
    ↓
Generate Prospect Brief

For each company, the search queries could include:

Find major announcements from Acme this month.

Find recent funding or acquisition news involving Acme.

Find evidence that Acme is expanding its engineering team.

Find products or features Acme launched recently.

Find recent partnerships or geographic expansion involving Acme.

Then only accounts with useful signals need to be surfaced to the rep.

That makes the workflow closer to:

100 Accounts
     ↓
Automated Research
     ↓
12 Accounts With Strong Signals
     ↓
Human Review
     ↓
Outreach

instead of manually researching all 100.

Adding Cloudsway Search API

One way to implement the retrieval layer is with Cloudsway Search API.

The workflow could look like:

CRM / Account List
       ↓
Company Query
       ↓
Cloudsway Search API
       ↓
Current Web Results
       ↓
Signal Classification
       ↓
LLM Analysis
       ↓
Prospect Brief

For example:

Find recent product launches, partnerships,
funding, hiring, or expansion news from Acme.

The retrieved results can then be passed into downstream logic that:

removes duplicate stories
identifies company events
categorizes sales signals
summarizes the evidence
preserves original sources
generates a short account brief

The separation of responsibilities is useful:

Cloudsway Search API
→ Discover current public information

Application
→ Filter, deduplicate, and structure

LLM
→ Interpret the signals

Sales Rep
→ Decide whether and how to reach out

Web search does not need to run the entire prospecting system.

It provides the live company context the rest of the workflow can use.

Keep Humans in the Final Step

There is an obvious temptation to go directly from:

Company Signal
      ↓
AI-Generated Message
      ↓
Send

I would avoid making that the default.

A better workflow is:

Search
   ↓
Detect Signal
   ↓
Generate Research Brief
   ↓
Human Review
   ↓
Personalize Outreach

Why?

Because a public signal does not automatically mean:

the company has the problem you solve
the specific prospect owns that problem
the timing is appropriate
the signal should be mentioned directly

Search can reduce research time.

The rep should still make the sales judgment.

Final Thoughts

Sales prospecting often has two separate problems:

1. Who should I contact?

2. Why might now be relevant?

Sales databases are strong at the first.

Current web research can help with the second.

A practical system might look like this:

CRM / Sales Database
        +
   Web Search
        ↓
Current Account Context
        ↓
Signal Classification
        ↓
Prospect Brief
        ↓
Human Review
        ↓
Outreach

The goal is not to generate more information about every prospect.

It is to surface a small number of recent, relevant signals that help sales teams decide where deeper research and outreach are actually worth their time.

How to Build a Competitor Monitoring Workflow with a Web Search API

Marcus ma — Thu, 13 Aug 2026 02:49:19 +0000

How to Build a Competitor Monitoring Workflow with a Web Search API

TL;DR

A useful competitor-monitoring system should not alert you every time a website changes.

It should detect meaningful business signals such as:

pricing changes
product launches
integrations
partnerships
funding
positioning changes
market reactions

A simple architecture looks like this:

Competitors
    ↓
Monitoring Objectives
    ↓
Web Search API
    ↓
Relevant Sources
    ↓
Deduplicate & Group
    ↓
Classify & Analyze
    ↓
Competitive Brief

The key distinction:

Page monitoring → What changed on a page we already know?

Web search → What new information appeared around this competitor?

Using both gives you much better coverage.

The Problem With Traditional Competitor Monitoring

Imagine a competitor launches a new enterprise plan on Monday.

On Tuesday, customers start discussing it.

On Wednesday, an industry publication covers it and the company announces a new integration.

If your process looks like this:

Open competitor websites every Friday

you will probably find the update eventually.

But you miss the context around it.

A useful monitoring system needs to answer:

What changed?

Why does it matter?

That second question is what turns website monitoring into competitive intelligence.

Step 1: Monitor Signals, Not Every Page

One of the easiest ways to build a noisy monitoring system is to watch everything.

A footer update and a major pricing change are both technically "changes."

They clearly do not have the same value.

Instead, define the signals you care about first.

Pricing

Examples:

new pricing plans
price increases or decreases
usage limits
enterprise packages
discounts
trial changes

Product

Look for:

new features
product launches
beta programs
integrations
new use cases

Technical

Useful signals include:

changelog updates
API changes
SDK releases
migration guides
documentation updates

Company

Examples:

acquisitions
funding
partnerships
leadership changes
geographic expansion

Positioning

Look for changes in:

target audience
category positioning
messaging
industries served
enterprise vs SMB focus

Now your monitoring objective becomes:

Tell me when something happens that could affect
our product, sales, marketing, or strategy.

Instead of:

Tell me whenever competitor.com changes.

Step 2: Separate Page Monitoring From Web Search

Page monitoring and web search solve different problems.

	Page Monitoring	Web Search
Input	Known URL	Research question
Best for	Pricing, docs, changelogs	News, launches, reactions
Main question	What changed here?	What is happening around this competitor?
Coverage	Fixed pages	Open web
Discovery	Limited	High

If you already know a competitor's pricing page:

https://competitor.com/pricing

monitoring that URL makes sense.

But you probably do not know the URL of:

tomorrow's partnership announcement
a new product page
an analyst report
a customer discussion
a news article about the company

Those require discovery.

A more complete system therefore looks like:

Known URLs
    ↓
Page Monitoring

Open Web
    ↓
Web Search

Step 3: Turn Competitive Questions Into Search Queries

The simplest approach is keyword monitoring:

[competitor] pricing

[competitor] launch

[competitor] partnership

[competitor] integration

[competitor] funding

That works, but it can produce a lot of irrelevant results.

For AI-oriented workflows, I prefer defining the underlying research objective.

For example:

Find recent changes to Acme's pricing,
packaging, or enterprise plans.

Or:

Find products or features Acme announced this month.

You can also search for reactions:

Find customer and industry reactions
to Acme's latest product launch.

Or strategic changes:

Find recent evidence that Acme
is moving toward enterprise customers.

This matters because competitive intelligence usually needs context, not just keyword matches.

Step 4: Use Search as the Discovery Layer

Once you have your monitoring objectives, you can run them through a Web Search API.

For example:

Scheduled Job
     ↓
Competitor
     ↓
Search Objectives
     ↓
Web Search API
     ↓
Relevant Sources

Suppose you monitor five competitors.

For each one, your workflow might search for:

Pricing
Product launches
Integrations
Partnerships
Funding
Company news
Market reactions
Positioning changes

At this point, however, you still do not have competitive intelligence.

You only have search results.

Step 5: Turn Search Results Into Events

Suppose Acme launches a new AI product.

Your search might return:

Acme product page
Acme company blog
Tech publication
Industry newsletter
Reddit discussion
Analyst article

If your system sends six alerts, users will quickly start ignoring them.

Instead, group those sources into a single competitive event.

For example:

Competitor: Acme

Signal:
Product Launch

Detected:
August 11

Event:
New enterprise AI search product

Sources:
- Product page
- Official announcement
- Industry coverage

Market Reaction:
Early discussion focuses on enterprise
security and pricing.

Why It Matters:
Acme appears to be moving further into
enterprise accounts.

Now the output is much more useful.

The unit you care about is not the URL.

It is the event.

Step 6: Deduplicate Related Sources

A basic processing pipeline might look like this:

Search Results
      ↓
Normalize URLs
      ↓
Extract Entities
      ↓
Compare Titles / Content
      ↓
Group Related Results
      ↓
Create Event

Possible grouping signals include:

competitor name
event date
product name
shared entities
similar titles
semantic similarity

This reduces duplicate alerts while keeping multiple supporting sources.

Step 7: Classify the Event

Once related sources are grouped, the application can classify the event.

For example:

{
  "competitor": "Acme",
  "category": "product_launch",
  "detected_at": "2026-08-11",
  "importance": "high",
  "summary": "Acme launched a new enterprise AI search product.",
  "sources": [
    "official announcement",
    "product page",
    "industry article"
  ]
}

Useful categories might include:

pricing
product
technical
integration
partnership
funding
positioning
market_reaction

Structured events are much easier to store, filter, compare, and analyze later.

Step 8: Let an LLM Analyze the Event

Once the evidence is grouped and structured, an LLM can do what it is good at:

summarize the change
compare it with previous events
identify strategic implications
explain why the signal matters
generate a concise brief

The architecture becomes:

Web Search
    ↓
Retrieve Evidence

Application
    ↓
Normalize + Deduplicate

LLM
    ↓
Analyze + Summarize

Dashboard / Slack / Email
    ↓
Deliver Intelligence

Keeping these responsibilities separate makes the workflow much easier to control.

Example: Daily Competitor Monitoring

Imagine we want to monitor Acme every morning.

The workflow could run:

Find recent pricing or packaging changes from Acme.

Find products or features Acme announced this week.

Find recent partnerships or integrations involving Acme.

Find recent market reactions to Acme's latest product launch.

Then process the results:

Daily Scheduler
      ↓
Search Queries
      ↓
Web Search API
      ↓
Current Results
      ↓
Deduplicate
      ↓
Classify
      ↓
LLM Analysis
      ↓
Competitive Brief

The resulting brief might look like:

Acme — Product Launch

Acme launched a new enterprise-focused AI search product this week.

Its product page emphasizes security, centralized administration, and enterprise integrations.

Several industry sources are already comparing it with existing enterprise search products.

Why it matters: This may signal a stronger push toward enterprise buyers and could affect positioning in security-sensitive sales opportunities.

That is much more actionable than sending a list of six links.

Adding Cloudsway Search API

One way to implement the web discovery layer is with the Cloudsway Search API.

The architecture stays the same:

Scheduled Workflow
       ↓
Competitor Query
       ↓
Cloudsway Search API
       ↓
Current Web Results
       ↓
Deduplicate
       ↓
Classify
       ↓
LLM Analysis
       ↓
Competitive Brief

Cloudsway handles the live web retrieval step.

The rest of the application can remain responsible for:

scheduling
historical storage
deduplication
event classification
scoring
LLM analysis
alerts
dashboards

This boundary is useful.

The search layer only needs to answer:

What relevant information appeared on the web?

Your application can then answer:

Is this new?

Is it important?

What does it mean?

Page Monitoring + Web Search

For a more complete system, combine both.

                Competitor
                    ↓
                  Router
                ↙        ↘
         Known Pages      Open Web
             ↓               ↓
       Page Monitor       Web Search
                ↘         ↙
             Event Processing
                    ↓
          Deduplicate & Classify
                    ↓
              LLM Analysis
                    ↓
            Competitive Brief

Use page monitoring when you already know where the information should appear.

Use web search when you do not.

Final Thoughts

A good competitor-monitoring system should not produce more alerts.

It should produce fewer, more meaningful signals.

A practical workflow looks like this:

Define Signals
     ↓
Search the Web
     ↓
Collect Evidence
     ↓
Group Related Sources
     ↓
Classify Events
     ↓
Explain Why They Matter

Page monitoring tells you what changed on sources you already know.

Web search helps discover what is happening outside that fixed list.

Combining the two turns competitor monitoring from a collection of page-change alerts into a much more useful competitive intelligence workflow.

Web Search API for AI Developers: How It Works and When to Use One

Marcus ma — Thu, 13 Aug 2026 02:08:29 +0000

TL;DR

A Web Search API gives AI applications programmatic access to current information from the web.
It is useful when an LLM needs information that changes frequently, such as news, pricing, documentation, product launches, or company updates.
A typical workflow looks like:

User Query
   ↓
Web Search API
   ↓
Relevant Web Results
   ↓
LLM / Agent / RAG
   ↓
Source-backed Answer

Search APIs, SERP APIs, and web scraping solve different problems.
For AI applications, the most important things to evaluate are relevance, freshness, structured output, source traceability, latency, and cost.

The Problem: LLM Knowledge Isn't Always Current

LLMs are great at explaining concepts, summarizing information, and reasoning over existing knowledge.

But things become harder when the question depends on information that changes frequently.

For example:

What changed in this API last week?

What products did this company launch this month?

How much does this service cost today?

What happened in the market this morning?

These aren't really "memory" problems.

They're retrieval problems.

If an AI application needs current information, it needs a way to retrieve that information while the task is running.

One common solution is a Web Search API.

What Is a Web Search API?

A Web Search API allows an application to search the web programmatically.

Traditional search is designed for humans:

User
 ↓
Search Engine
 ↓
Search Results Page
 ↓
Open Pages
 ↓
Read Information

With an API, the workflow becomes machine-readable:

Application
 ↓
Web Search API
 ↓
Structured Search Results

Instead of rendering a search results page, the API can return data such as:

{
  "title": "Example Page",
  "url": "https://example.com/article",
  "snippet": "Relevant information from the page..."
}

An application can then pass those results directly into an LLM or another processing step.

That makes search part of the AI workflow itself.

How Does a Web Search API Work?

At a high level, the process is straightforward.

Query
  ↓
Search Processing
  ↓
Ranked Results
  ↓
Structured Response
  ↓
AI Application

1. Generate a search query

The query may come directly from the user:

latest AI search infrastructure announcements

Or an AI agent may generate it automatically while working on a larger task.

For example:

User: Research recent developments in AI search infrastructure.

Agent:
1. Search recent company announcements
2. Search product launches
3. Search industry news
4. Compare findings
5. Generate report

Search becomes one tool inside a larger reasoning loop.

2. Search the web

The search layer finds pages related to the query.

Depending on the task, relevance alone may not be enough.

For a query like:

OpenAI API pricing

an old result could still be highly relevant while being completely useless for answering a question about current pricing.

For AI systems working with changing information, freshness matters alongside relevance.

3. Rank the results

The search API returns the most useful results instead of requiring your application to process thousands of pages.

This matters because everything you pass to an LLM has a cost.

Bad retrieval means:

Irrelevant Results
        ↓
More Context Tokens
        ↓
More Noise
        ↓
Worse Generation

Good search quality improves the entire downstream pipeline.

4. Return structured data

A useful search response might include:

[
  {
    "title": "Company Announces New Search Product",
    "url": "https://example.com/news",
    "snippet": "The company announced...",
    "published_at": "2026-08-01"
  }
]

Structured output makes it much easier to:

filter results
rank sources
extract URLs
generate citations
pass evidence to an LLM

5. Let the AI reason over the results

Search shouldn't necessarily generate the final answer.

A clean architecture separates retrieval from reasoning:

Search API
   ↓
Find relevant evidence

LLM
   ↓
Compare, summarize, and reason

Application
   ↓
Present the final answer

This separation is particularly useful when building agents and RAG systems.

Why Not Just Ask the LLM?

Imagine you're building a competitor-monitoring agent.

The user asks:

What has Company X launched in the last 30 days?

Without web retrieval:

Question
   ↓
LLM Knowledge
   ↓
Answer

The model may not have access to those announcements.

With search:

Question
   ↓
Search Recent Web Sources
   ↓
Retrieve Announcements
   ↓
LLM Analyzes Results
   ↓
Answer + Sources

The model no longer needs to "know" everything beforehand.

It only needs to reason effectively over the evidence you retrieve.

This is one of the most useful patterns for building web-connected AI applications.

Web Search API vs SERP API vs Web Scraping

These tools are often grouped together, but they solve different problems.

	Web Search API	SERP API	Web Scraping
Main goal	Find relevant information	Retrieve search engine results	Extract content
Starting point	Information need	Search query	Known URL
Typical output	Relevant web sources	Rankings, URLs, snippets	Page content
Common use	AI, RAG, agents, research	SEO and rank tracking	Data extraction

SERP API

A SERP API is useful when the search results themselves are the data you care about.

For example:

Which pages rank for "AI search API"?

What position does my website appear in?

Which domains dominate this SERP?

That makes SERP APIs especially useful for SEO tools.

Web Search API

A Web Search API is useful when you're trying to answer:

Where can I find useful information about this question?

The goal is retrieval rather than analyzing the SERP itself.

Web Scraping

Scraping usually starts after discovery.

You already know the URL:

https://example.com/pricing

Now you want to extract specific information from that page.

A useful mental model is:

Search → Find the page
Scraping → Extract from the page

Many real-world applications use both.

Where Web Search Fits Into RAG

Traditional RAG often looks like this:

Documents
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Database
   ↓
Retrieval
   ↓
LLM

This works well when the information already exists in your indexed knowledge base.

But what happens when the answer exists only on the public web?

Or when the information was published ten minutes ago?

That's where web search can complement vector retrieval.

User Question
      ↓
   Router
    ↙   ↘
Internal   Current Web
Knowledge  Information
   ↓           ↓
Vector DB   Web Search
    ↘         ↙
       LLM
        ↓
      Answer

You don't necessarily need to replace your vector database.

Web search can become another retrieval source.

Example: Building a Research Agent

Suppose the user asks:

Research recent developments in AI search infrastructure.

A simple agent might perform:

1. Search recent industry news
2. Search company announcements
3. Search product documentation
4. Review retrieved sources
5. Identify important developments
6. Search again for missing information
7. Generate the final research brief

The interesting part is step 6.

An agent doesn't always search once.

It might run a loop like:

Search
  ↓
Review Evidence
  ↓
Enough Information?
  ├── Yes → Generate Answer
  └── No  → Generate New Query
               ↓
             Search

This makes search especially useful for agentic workflows.

Common Use Cases

1. AI Agents

Agents can use search as an external information tool.

Examples include:

company research agents
market research agents
news monitoring agents
technical research agents
competitive intelligence agents

2. RAG Systems

Web search can provide information that hasn't yet been added to your internal knowledge base.

A common architecture is:

Internal information → Vector retrieval

Current public information → Web search

3. Research Tools

Search APIs can help discover:

research papers
reports
documentation
company announcements
industry analysis
technical resources

The LLM can then organize and synthesize those sources.

4. Competitive Intelligence

A competitor-monitoring workflow might search for:

Competitor pricing changes
Competitor product launches
New partnerships
Funding announcements
Company news
Documentation updates

Search handles discovery.

Your application handles classification, comparison, and analysis.

What Should You Look for in a Web Search API?

Not every search API works equally well for AI applications.

Here are the main things I'd evaluate.

Relevance

If the search results don't match the user's intent, everything downstream gets worse.

Bad Retrieval
   ↓
Bad Context
   ↓
Bad Answer

Retrieval quality matters as much as model quality.

Freshness

Fresh results are critical when working with:

news
pricing
documentation
product updates
company announcements
market information

For these tasks, a highly relevant page from two years ago may still be the wrong result.

LLM-Ready Output

The easier the results are to process, the less infrastructure you need around them.

Structured responses can reduce additional:

parsing
HTML cleaning
extraction
transformation

before sending results to your model.

Source Traceability

For research-oriented applications, you usually want to preserve the source URL.

That allows your final system to produce something closer to:

Claim
 ↓
Evidence
 ↓
Original Source

instead of an answer that can't be verified.

Latency

Agents may search multiple times for a single user request.

For example:

Search
 ↓
Analyze
 ↓
Search Again
 ↓
Analyze
 ↓
Generate

A few hundred milliseconds of additional latency per search can accumulate quickly.

Cost

The same applies to cost.

Don't evaluate search pricing only as:

cost per API request

Think about:

searches per user task × requests × user volume

The workflow-level cost is what ultimately matters.

Using Cloudsway Search API as the Retrieval Layer

One option for this architecture is the Cloudsway Search API.

The basic pattern is:

User / Agent Query
       ↓
Cloudsway Search API
       ↓
Structured Web Results
       ↓
LLM / Agent / RAG
       ↓
Final Response

For example, an agent could start with:

Find recent developments in AI search infrastructure.

The search results could then be passed to an LLM to:

summarize recent developments
compare companies
identify product announcements
extract important changes
generate a source-backed research brief

The useful architectural point here is that the responsibilities remain separated:

Cloudsway Search API
→ web discovery and retrieval

LLM
→ reasoning and synthesis

Your application
→ workflow and user experience

So you don't need to build a full web search layer before adding current web information to an AI application.

Final Thoughts

Web Search APIs solve a simple but important problem:

How does an AI application access information that changes after the model was trained?

A common answer is to retrieve that information at runtime.

User Question
      ↓
Web Search
      ↓
Current Evidence
      ↓
LLM Reasoning
      ↓
Source-Backed Answer

This pattern works especially well for:

AI agents
RAG systems
research applications
competitive intelligence
any AI product that depends on current public information

When choosing a Web Search API, I'd focus less on the number of results it can return and more on how well it works inside the complete AI workflow.

The key questions are:

Are the results relevant? Are they fresh? Can I trace the sources? Can my LLM consume them easily? And what happens to latency and cost when an agent searches multiple times?

Those factors usually matter much more once you move from a demo to a real AI application.

welcome！

Marcus ma — Thu, 06 Aug 2026 07:31:27 +0000

Marcus ma

Aug 6

How to Add a Real-Time Search Layer to an Agent Graph

#ai #agents #api #webdev

7 min read

How to Add a Real-Time Search Layer to an Agent Graph

Marcus ma — Thu, 06 Aug 2026 05:32:27 +0000

Marcus ma

Aug 6

How to Add a Real-Time Search Layer to an Agent Graph

#ai #agents #api #webdev

7 min read