DEV Community: techpotions

SearXNG JSON API for LLM Grounding

techpotions — Tue, 04 Aug 2026 21:37:00 +0000

The first time a large language model invented a statistic in my draft about search APIs, I knew it was time to ground the thing against real web results. A self-hosted SearXNG JSON API now sits between the model and the internet for our blog pipeline, replacing a paid :online markup and costing nothing at the volume we run.

What matters more than free: we designed the integration so a search failure degrades the output gracefully rather than breaking the pipeline. That is the piece most builders skip, and it is the piece that makes self-hosting safe to depend on overnight.

SearXNG JSON API Endpoint

SearXNG exposes a single /search endpoint. You tell it you want structured output with format=json, and it hands back a clean payload instead of a rendered results page. The request below is the exact call our pipeline makes—no middleware, no abstraction.

GET https://searxng.techpotions.app/search?format=json&q={query}&categories={categories}&time_range={time_range}

Parameters that earn their keep

Not every parameter matters for LLM grounding. These are the three we actually use in production.

Parameter	Purpose	Example
`q`	The search query—exactly what you would type into a search box	`q=vector databases compared 2025`
`categories`	Scopes the engine set. `general` hits broad web results; mix with `news`, `science`, or `files` when the query calls for it	`categories=general,news`
`time_range`	Narrows results by recency. Critical when grounding the model against what is ranking right now	`time_range=month`
`format`	Must be `json`. Without it, you get an HTML page you do not want to parse	`format=json`

time_range accepts day, week, month, or year. For our blog pipeline, month is the default. When a query targets a fast-moving topic, we drop to week. When it is evergreen, year keeps the signal broad.

The raw TypeScript call

async function searchSearXNG(query: string, timeRange = "month"): Promise<SearchResult[]> {
  const base = "https://searxng.techpotions.app/search";
  const params = new URLSearchParams({
    format: "json",
    q: query,
    categories: "general",
    time_range: timeRange,
  });

  const response = await fetch(`${base}?${params}`);
  if (!response.ok) return [];

  try {
    const data = await response.json();
    return (data.results ?? []).slice(0, 10);
  } catch {
    return [];
  }
}

What It Replaced: Paid Search Markup

Before the SearXNG instance, our blog pipeline used OpenRouter's :online search markup. The model would tag a prompt with :online, and OpenRouter would inject web results from a paid search backend. It worked. It also cost money on every query and leaked portions of our content roadmap into a third-party search API.

That privacy piece is not theoretical. Our pipeline surfaces what is currently ranking for a target query so drafts are framed against the real SERP rather than the model's stale training data. The search queries are literally a list of the topics we plan to publish over the next quarter. Keeping those queries inside infrastructure we own matters.

Self-hosting SearXNG makes the grounding layer free at the volume we run, but the real win is that our content roadmap stays ours. For AI services where the prompts themselves are proprietary, this pattern applies just as directly.

The Resilience Pattern: Degrade, Don't Fail

Search is an enhancement to generation, not a precondition for it. A draft written without fresh sources is slightly worse. A draft that never gets written because a cron job exploded at 3am is an editor staring at an empty CMS in the morning.

Our searchSearXNG wrapper never throws. Three failure modes all converge on the same outcome: an empty array.

Non-OK response — upstream engine timed out, instance is restarting, network blip
Network error — DNS failure, box is unreachable, TLS expiry
Malformed JSON — upstream returned an error page, or a reverse proxy injected something unexpected

Every path returns []. The caller receives zero results instead of an exception in the stack. If your grounding call can throw, your nightly generation job now has a hard dependency on a service you self-host on hardware you are not watching at 3am.

This is the decision worth the whole integration. Most AI pipelines wire up a search API and then wrap it in a retry loop, as though a retry will fix an instance that has been down for four hours. Retries are for transient failures. This pattern handles the persistent ones.

// No try/catch at the call site. No retry loop. No alerts.
const sources = await searchSearXNG(query);
// sources is always an array. Could have 10 results. Could have 0.
// Build the prompt either way.
const groundedPrompt = buildPrompt(query, sources);

Two Jobs the SearXNG JSON API Handles

1. SERP-aware framing

The first call in the pipeline searches the target query and returns what is ranking right now. That surface shapes the draft outline: which angles the top results take, what headings they use, what questions they answer. The model sees the real competitive landscape instead of guessing from stale weights.

2. Source grounding

The second call searches for authoritative sources on each factual claim the draft makes. The model receives real URLs and real snippets and weaves them into the prose. This dramatically reduces hallucinated statistics and invented quotes. We wrote about the broader pipeline approach in stopping AI blog pipeline hallucinations.

The Honest Caveat

SearXNG is a metasearch aggregator. It does not crawl the web itself. Each query fans out to upstream engines—Google, Bing, DuckDuckGo, and others depending on configuration—and those engines rate-limit. Some block datacenter IPs. Result quality fluctuates in a way a paid search API with a dedicated crawl index does not.

You own the uptime. You configure the instance, you watch the logs, you handle the inevitable day an upstream engine changes its response format and breaks result parsing. This is the right trade when grounding is a nice-to-have enhancement to generation. It is the wrong trade when search results are the product.

Spend the dollars when every result matters. Self-host when the fallback is a perfectly functional model prompt without web context.

Getting SearXNG Running

SearXNG ships as a Docker image with a single container. Bring a domain, add a Let's Encrypt reverse proxy, and set the SEARXNG_SECRET environment variable to a random value for encryption. The official docs cover the compose file.

Our instance at searxng.techpotions.app runs behind a Traefik reverse proxy with automatic TLS. The configuration file enables JSON output by default and limits the engine set to the ones that reliably return results from the region our queries target.

For builders integrating search into AI products, we offer AI consulting and development services that include grounding-layer architecture. If you are starting from scratch, our project launch framework covers the patterns we reuse across client builds.

FAQ

What is the SearXNG JSON API?

It is the structured output mode of a self-hosted SearXNG metasearch instance. You set format=json on the /search endpoint, and the server returns a JSON object with a results array containing titles, URLs, snippets, and engine metadata—no HTML parsing required.

Is the SearXNG JSON API free?

Yes. The software is open source. You pay only for the infrastructure you run it on. At modest query volumes, a small VPS handles the load without additional search API costs. The trade is that you own the operational burden: uptime, engine configuration, and the occasional upstream breakage.

Why use SearXNG instead of a paid search API for AI agents?

Two reasons make the self-hosted path compelling. First, search queries stay private on your infrastructure—relevant when those queries reveal your content roadmap or proprietary prompts. Second, the resilience pattern of degrading to an empty result set rather than throwing means a search outage does not fail the generation job. Paid APIs solve the uptime problem; they do not automatically solve the architectural one.

Can I use the SearXNG JSON API in production?

Yes, with the right failure design. Treat search results as an optional enhancement, not a required input. When results are available, the output is stronger. When the instance is down or upstream engines block, the pipeline runs anyway. That architecture is what makes self-hosting production-safe for overnight automation.

When to Turn Off Next.js Image Optimization

techpotions — Tue, 04 Aug 2026 21:28:41 +0000

Turning on next.js images unoptimized looks like a mistake in a code review—and in our own next.config.ts, it’s exactly what we did, with a comment so nobody reverts it. Our covers were getting softer on mobile, and the culprit was stacking two lossy encoders. Here’s when that config is correct, when it’s dangerous, and how to document it so the next developer doesn’t “fix” it.

When `next.js images unoptimized` is the correct call

The safe time to set images.unoptimized is when your upstream source already delivers a well‑sized, modern‑format image. In our case, Payload CMS re‑encodes every upload to WebP once, at upload time. Next.js’s <Image> component then wanted to run its own lossy transformation on top of that already‑lossy WebP. The result: double encoding that visibly softened photographic covers on mobile.

Turning the optimizer off let the CMS‑encoded WebP serve as‑is. The covers got sharper immediately. The config that made it happen lives in the repo with a comment that explains the trade‑off:

// next.config.ts
images: {
  unoptimized: true, // Payload already encodes to webp at upload;
                     // Next's optimizer was re-encoding and softening images.
                     // Keep off.
}

The broader rule is simple: image optimization is not additive. Each lossy pass compounds the artifacts of the previous encode. If something upstream already produced a compressed, correctly sized image in a modern format (WebP, AVIF), a second optimizer is not making it better—it’s re‑encoding the artifacts of the first encode. Know how many lossy passes your pipeline runs before you let Next.js add another one.

When you must keep Next.js image optimization enabled

Keep Next.js’s optimizer on when you are serving original‑quality uploads—untouched JPEGs, PNGs, or even large TIFFs that come straight from a user’s camera or a design tool. Those files are almost never ready for the web. The optimizer generates responsive srcset variants across multiple viewport widths, converts to modern formats when the browser supports them, and compresses the output—all without you building a media pipeline.

You also need the default optimizer when your source images are not already correctly sized. If your CMS just stores whatever the editor uploaded, turning unoptimized on means you’ll be serving 4000‑pixel‑wide hero shots in a 300‑pixel card. That’s a performance disaster.

Finally, remember that images.unoptimized is a global flag. Once you flip it, you assume responsibility for dimensions, format, and art direction. If your CMS doesn’t handle that, you’ve simply removed optimization rather than avoided duplication. A naive reading of this article will make somebody’s site worse—make sure your pipeline is actually doing the job before you reach for the off switch.

On Vercel, image optimization is a billed transformation. Turning it off where it adds nothing removes a cost line as well as a quality problem. That’s a nice side effect, but quality was our primary driver. Don’t let the cost tail wag the quality dog.

How to configure `next.js images unoptimized` (and why it needs a comment)

The configuration itself is a single key in next.config.js or next.config.ts:

module.exports = {
  images: {
    unoptimized: true,
  },
}

That’s it. But without context, it reads like a lazy shortcut. Any developer—or a future you—will see it and wonder if it was left over from a debug session. A comment transforms it from a potential bug into a documented decision. Ours spells out the CMS workflow, the double‑encoding problem, and the quality improvement. If you’re adopting this pattern, steal that format.

If you need more granular control (e.g., bypassing optimization only for images from a specific CDN), you’ll have to build a custom loader. The global flag is intentionally blunt. For most sites where the CMS already handles the heavy lifting, that bluntness is a feature—it keeps the pipeline simple and predictable.

AI Automation Agencies: Real vs Repackaged

techpotions — Mon, 03 Aug 2026 18:02:02 +0000

We build AI automation at techpotions, so I’m biased. The only way this article earns your trust is by being so specific that even an agency looking for shortcuts couldn’t replicate it. If you’re evaluating ai automation agencies today, it’s brutally hard to tell the operators who’ve fought real production fires from the repackagers who’ve only ever run demo workflows. Below are the questions we ask ourselves — the uncomfortable ones — and the answers that separate the real from the repackaged.

How to separate genuine AI automation agencies from resellers

The quickest test: ask what the agency itself runs and what broke. A shop that has never scraped its knees has no stories. At techpotions, our publishing pipeline is a Payload post.published webhook into n8n. The workflow fans a new article out to dev.to, Hashnode, and LinkedIn, sets canonical URLs back to our site, then writes the resulting platform URLs back into the Payload database. It broke in production — syndicated copies went out with relative links, so /services/ai resolved to dev.to/services/ai and died. We fixed it by absolutising every link before posting. An agency that can’t tell you something similar has only ever watched, not run.

Ask where the human approval gate lives

Ethical ai automation agencies don’t remove the human from decisions that affect your brand. In our pipeline, generation and review stay inside the application — a person approves every draft. Only after approval does n8n cross-post. That architecture lets us add or remove platforms with zero code change, but the approval gate is immutable. If an agency advertises fully automated end-to-end social or content, they’re selling you unreviewed output. The question isn’t whether they “have a review step,” but where it sits. Ours is in the application, not in the automation. That’s what protects you from the drift that comes when no human sees the copy.

The real tell: what an AI automation agency refuses to automate

This cuts hard, even against us. We killed our own trending-news content mode entirely. It churned out posts that might as well have been spun — undifferentiated, zero proof points. We rebuilt the generator so it writes nothing unless a topic carries a real, verifiable detail the reader can check. That means it sometimes outputs silence. An agency that says “everything is automatable” hasn’t run anything long enough to hit the limits. When you interview ai automation agencies, ask what they’ve deliberately removed from their own automation. If the answer is “nothing,” keep looking.

Syndication doesn’t buy you backlinks — and honest agencies say so

Anyone packaging syndication as link building is either ignorant or dishonest. Syndicated links on dev.to, Hashnode, LinkedIn, and similar platforms are nofollow. They pass zero PageRank. What syndication actually does is buy referral traffic, brand reach, and visibility in AI-generated answers — all valuable, but nothing to do with backlinks. Watch for agencies that conflate the two. If they lead with “do-follow” or imply SEO juice, they’re repackaging someone else’s playbook without understanding the nofollow reality. Read more about how we think about building AI agents with n8n to see where real automation value lies.

What to look for in your next conversation

When you sit down with a potential automation partner, here’s a simple scorecard:

Question	What signals a real operator
“Show me an internal automation that broke.”	Specific story with the fix, not “We’ve never had a failure.”
“Where is the human approval gate?”	Inside the application, before any automated distribution (ours: blog → review → n8n cross-post).
“What will you refuse to automate for me?”	Clear boundaries, like no-touch content generation without a proof point.
“Does my syndication plan build backlinks?”	“No — syndication platforms use nofollow. You’ll get traffic, brand, and AI visibility.”

AI automation agencies that pass these four questions have earned the right to talk about their stack. The rest are selling packaging.

FAQ

How can I tell if an AI automation agency actually runs its own workflows?

A genuine operator will readily describe their own internal automations — including what broke and why. If they can’t share a specific failure story that required a fix, they’re likely running demos, not production systems.

Where should human review sit in an automation pipeline?

Ethical agencies insert a human approval gate before any automated action goes public. Ours lives inside the application: every draft is reviewed before n8n cross-posts it. End-to-end automation with no human review means unreviewed output is heading straight to your audience.

Do syndicated blog posts help with SEO backlinks?

Syndicated links on platforms like dev.to, Hashnode, and LinkedIn are nofollow. They won’t pass PageRank. Syndication buys referral traffic, brand reach, and visibility in AI-generated answers — not backlinks for SEO.

What Agentic AI Development Services Actually Ship

techpotions — Mon, 03 Aug 2026 18:01:37 +0000

When technical buyers go shopping for agentic AI development services, they usually picture the agent—the autonomous loop that plans, tools around, and completes a task. That is the glamorous 10%. The other 90% is everything that stops the loop from shipping a regression, returning unparsable sludge, or confidently breaking something expensive. Those ungroomed parts are what a serious quote is actually pricing.

At techpotions, we learned this the hard way, shipping agentic systems for recruitment screening, voice products, and content pipelines. Here is the deliverable list buyers never see quoted—and why each line item is real work.

What Agentic AI Development Services Actually Deliver

1. An Eval Suite That Ships Before the Agent

An agent without an eval set is a chemistry set without a scale. You cannot call a prompt change an improvement if you have no reproducible way to measure it against the last version. You will ship a regression, and you will not know until a customer tells you.

We ship the eval suite before launch. For one product, we built 380 golden cases; for another, 240. Every prompt edit runs the full suite. The cases cover happy paths, edge cases, adversarial inputs, and the product’s specific error modes (e.g., misreading years of experience, confusing similar company names).

Building those cases is the labour: extracting them from raw logs, neutralizing live data, writing the grade rubric for each one, and wiring the harness into CI so it blocks a deploy. This line item alone is often the heaviest in an agentic build, and it is conspicuously absent from most scoping conversations.

2. A Structured Output Contract, Not Free Text

An agent that returns free text is not integratable. Downstream systems—search indexes, CRMs, dashboards, notification triggers—need a contract. If the agent writes a novel, someone has to parse it, and that someone will hate you.

We use native json_schema strict mode wherever the provider supports it, with a json_object fallback for models that reject strict schemas. This matters because any agentic AI development stack that fans out across multiple model providers through a gateway cannot assume uniform structured-output support. You code for the weakest link.

The work here is not just declaring a schema. It is designing the schema so it survives real variability—nullable fields that triple-providers omit, enum sets that stay open, and validation that catches what the model got wrong before it hits a database constraint. The deliverable is a typed contract, a validator, and a retry loop that asks the model to fix its own broken JSON instead of silently dropping the turn.

3. A Defined Human Handoff Point

Decide what is expensive to be wrong about and put the human there. This is a design decision, not an afterthought, and it materially changes the system architecture.

In our recruitment screening product, the agent produces a ranked shortlist with quoted evidence per criterion—transcript snippets for each claim, scored against the job description. It never auto-rejects a candidate. The final “advance / decline” call stays with the recruiter, because a false negative is a breached commitment to the client. You can read the full case study here.

In our orientation product, we priced human review as a separate tier rather than baking it into the base service. Some organizations want fully automated onboarding flows; others need a human in the loop for compliance-sensitive steps. The architecture supports both without the overhead collapsing into a single, inflated per-seat cost.

The deliverable is an explicit handoff protocol: the trigger condition, the data packet the human reviews, the UI they review it in, and the timeout/fallback if no one responds. That all has to be built.

4. Grounding Infrastructure That Never Takes Generation Down

An agent answering from model memory answers from stale memory. Grounding it in fresh data is table stakes, but the implementation detail that separates a demo from a production service is what happens when the grounding layer fails.

We run a self-hosted SearXNG instance as the grounding layer for our content pipeline. The wrapper is written to never throw: on any failure—timeout, no results, malformed response—it returns an empty result set and logs the event. This is non-negotiable. A grounding search going down must not take generation down with it.

The deliverable is the retriever API with the circuit breaker, the cache layer that keeps the most-queried documents warm, and the fallback prompt that tells the model “you have no live context, respond conservatively” instead of letting it hallucinate into the gap.

5. Observability on the Ops Surface

Someone inside your customer’s organization will need to answer: what did our agent say to this specific person at this specific time? If the answer is “check the server logs,” you shipped a gap.

For the voice product we built, the dashboard covers calls, transcriptions, agent interactions, CRM records, and user management. It is a single pane because the people who answer those questions—support leads, compliance officers, product managers—do not live in your observability stack.

This surface is not a thin skin over telemetry. It requires logging every turn with the full payload (prompt, completion, tool calls, latency), indexing it on the dimensions the business cares about, and building a UI that can replay a conversation without looking like a raw JSON dump. None of that is glamorous. All of it is the difference between “the agent works” and “we can run a business on it.”

Why the Model Call Is the Small Part

The dominant framing in the market—and on cloud providers’ landing pages—is that you pick a framework, chain some tools, and deploy. AWS positions agentic AI as something that “removes constraints across how you build, secure, and ship software.” Google Cloud offers “a flexible and modular framework for developing and deploying AI agents.” These are useful raw ingredients. They are not the meal.

What is missing from the platform pitches is the work that happens between the framework and the production line: designing the guard evaluator, building the golden dataset, hardening the output schema, wiring the human fallback, and making the whole thing observable enough to trust. These are the deliverables an agentic AI development services engagement should quote line by line, because they are where the budget actually goes.

If you are scoping a build, start here with the checklist no one sends you, or read how we add evals to an LLM feature for the specific pattern.

FAQ

What do agentic AI development services actually cost?

We don’t publish day rates or fixed prices because the scope varies dramatically: a single-agent screening tool with a handoff point is a different build from a multi-agent voice product with a compliance dashboard. The cost is driven by the eval suite size, the number of integrations, and the observability surface. Reach out here with your use case, and we will map it to deliverables.

How long does it take to build an agentic AI product?

From our experience, a focused agentic product (e.g., a screening agent with a human review tier) takes weeks, not months, to reach a production-grade MVP—if the scope is tightly defined and the eval suite is built alongside the first prompt experiments. Larger multi-agent systems with custom dashboards and multiple integrations extend the timeline. The blocker is rarely the model logic; it is the grounding, the contracts, and the edge cases.

Do I need structured output for my agent?

Almost certainly yes. If your agent is consumed by a human reading text, free-form output can work. But if its decisions feed into another system—a database write, a status change, a notification trigger—you need a schema. Without it, you are pushing the parsing problem downstream, where it will surface as brittle regex, dropped records, and angry integration partners. We ship a structured output contract on every agentic build.

When Not to Use an AI Agent

techpotions — Mon, 03 Aug 2026 17:30:24 +0000

When not to use an AI agent isn’t about cost or complexity first — it’s about whether you can name every step from start to finish. We build AI agents for a living, yet the smartest moves in our own stack replaced would‑be agents with plain automation and human judgment. Here are the three times we killed the agent, why, and the decision rule that’ll keep you from over‑engineering.

When Not to Use an AI Agent

The industry defaults to reaching for an agent the moment work involves text, images, or decisions. But most tasks sold as “agent problems” really fall into two simpler buckets: deterministic workflows where the steps are known, and high‑stakes decisions where getting it wrong is expensive and invisible. In both, adding an LLM that decides its own path doesn’t improve outcomes — it introduces a failure mode you can’t enumerate.

Our litmus test, refined on our own production systems, is: use an agent only when the path is genuinely unknown at design time and the cost of a wrong step is recoverable. The three cases below show exactly where that line sits.

Case 1: Publishing Fan‑Out — When Every Step Is Known, Write a Pipeline, Not an Agent

When we publish a blog post, it gets pushed to dev.to, Hashnode, and LinkedIn — with canonical URLs pointing back to the original, and the resulting social URLs written into our database. This is a webhook into an n8n workflow that processes the payload, sends the content to each platform’s API, captures the response, and updates a CMS record.

There is no model in it. There shouldn’t be. Every single step has exactly one correct outcome. An agent that “decides” which API to call or how to format the payload would only add a way for it to be wrong — a malformed LinkedIn post, a missing canonical tag, a database write that fails silently because the agent guessed a different field name.

The rule: If you can write down the sequence — authenticate, post, parse response, write back — you don’t need something that invents the sequence at runtime. Deterministic code is faster, cheaper, and trivially testable.

Case 2: Content Ideation — Constrain the System, Don’t Optimize the Agent

The obvious agentic design for a content generator is a loop: ideate topics, judge them with an LLM, pick the best, and start writing. We built a version close to that and killed it.

Why? Because it defaulted to trending news. The agent dutifully recycled existing top‑ranking pieces into summaries that added no net‑new signal. It wasn’t “hallucinating” — it was rationally producing the kind of content that already existed, just faster. That’s not what we needed.

The replacement isn’t smarter; it’s dumber and better. The generator now reads a human‑curated backlog of real proof points and first‑hand notes. It drafts only topics that carry a verifiable insight from our own work. If the backlog is empty, it writes nothing at all. No agent loop, no ranking model — just a constraint that the input must contain a concrete, non‑searchable claim.

Constraining the system beat improving the agent. When the universe of acceptable inputs is narrow and deterministic, you don’t need an LLM to filter; you need a rule.

Case 3: Candidate Screening — Automate the Reading, Not the Judgment

In our candidate screening, a model reads and scores applications. That part is worth automating — turning unstructured text into structured signals saves hours. But the final decision? That stays with a human.

A false reject here is expensive and invisible: you’ll never know you missed the best hire, and the candidate gets no meaningful feedback. Automating the reading was a clear win; automating the judgment was not. The model produces a score, a human reviews the shortlist, and the human decides.

This is the third spot where agents break down: when the cost of a wrong step is asymmetric and hard to detect, keep the human in the loop. A model can recommend; it shouldn’t decide.

The Decision Rule: Agent, Deterministic Code, or Human

Distilled from those three production decisions, here’s the heuristic we apply to every new feature:

Situation	Solution
You can enumerate every step and each step has one correct answer	Deterministic code (webhooks, serverless functions, n8n)
The path is ambiguous at design time, but wrong steps are cheap and recoverable	AI agent (LLM‑driven decision loops)
The path is ambiguous or high‑stakes, and wrong steps are expensive or invisible	Human judgment, optionally supported by deterministic scoring

When in doubt, ask: “If this step goes wrong, will we notice immediately?” If the answer is no, don’t give it to an agent.

When Selling Agents Means Telling Clients “Don’t Build One”

We offer AI consulting and agent development. The honest version of this business — the one that builds trust and actually works — is sitting down with a client who walks in with three agent ideas and saying, “Number two doesn’t need an agent. We can build it as a plain automation in a couple of days for a fraction of the cost.”

The best outcome isn’t a flashy autonomous system; it’s a pipeline that runs silently and never breaks. Most of the value we deliver is recognizing when not to use an AI agent, and building the other two ideas as reliable code. If you want to have that conversation about your own stack, start here.

FAQ

What is the simplest test for when not to use an AI agent?

If you can enumerate every step and there is exactly one correct outcome for each, use deterministic code. If the cost of a wrong step is expensive and hard to detect (e.g., rejecting a great candidate), keep a human in the loop. Only use an agent when the path is genuinely unknown at design time and the cost of a mistake is recoverable.

How is an AI agent different from a chatbot, and why does that matter?

A chatbot follows a fixed conversation flow; an AI agent decides its own sequence of actions. Agents introduce uncertainty — exactly what you want to avoid when you already know the right steps.

Can I use n8n instead of building a full AI agent?

Absolutely. For deterministic workflows like our publishing fan‑out, we use n8n without any LLM step. It’s faster, cheaper, and never hallucinates a URL. You don’t need an agent just because a tool supports one.

Shiki in Next.js App Router: Zero Client JS

techpotions — Mon, 03 Aug 2026 17:29:55 +0000

Server-side syntax highlighting with Shiki in the Next.js App Router means you ship exactly zero client‑side JavaScript for code blocks. That’s 0 KB of grammar definitions, 0 KB of theme data, and 0 KB of a highlighter library—browser downloads that often add 50–100 KB before a single line becomes readable. Instead, every code block is rendered into coloured HTML spans on the server, and the browser just paints it. This how‑to walks through the core move, the performance details that make it safe, and the maintenance trap that will bite you if you wire it into a CMS.

Shiki + Next.js App Router: Zero KB of Client JS

The takeaway: when you call Shiki’s codeToHtml inside a Server Component, the visitor’s browser receives ready‑to‑display HTML. No highlighter bundle is ever downloaded. Compare that with a typical client‑side highlighter like Prism or Highlight.js, where the grammar and theme data are part of the bundle the browser must fetch, parse, and execute before code becomes readable. On a slow connection, that delay is visible—a flash of unstyled text or a blank block while the scripts load.

With Shiki in the App Router, that bundle simply doesn’t exist. The server does the work once at render time, and the output is <span style="color: #..."> elements that any browser can handle immediately. The performance win is permanent, not just a first‑load trick.

The Core Move: Highlighting in a Server Component

Create a highlight function that runs codeToHtml and returns the HTML string. The key is an allowlist of supported languages that fall back to plain text when an unsupported language is requested. Without that fallback, one editor picking an unlisted language will take down the page render.

Here’s the function we use at techpotions to highlight every code block on our blog:

import { codeToHtml } from 'shiki';

const ALLOWED_LANGUAGES = new Set([
  'typescript', 'tsx', 'javascript', 'jsx',
  'json', 'bash', 'python', 'go', 'rust',
  'sql', 'html', 'css', 'yaml', 'markdown',
  'text'
]);

export async function highlight(code: string, lang: string): Promise<string> {
  const safeLang = ALLOWED_LANGUAGES.has(lang) ? lang : 'text';
  return codeToHtml(code, { lang: safeLang, theme: 'vitesse-dark' });
}

Call this function inside a Server Component and inject the result with dangerouslySetInnerHTML. Because the HTML is generated on the server, there’s no hydration mismatch and no client‑side JavaScript involved.

// app/(blog)/[slug]/page.tsx
import { highlight } from '@/lib/shiki';

export default async function BlogPost({ code, language }) {
  const html = await highlight(code, language);
  return <div dangerouslySetInnerHTML={{ __html: html }} />;
}

Why This Approach Scales: Shiki’s Internal Caching

A common worry: “If I highlight 20 code blocks on a page, will I create 20 highlighter instances?” No. Shiki maintains a singleton highlighter internally. It lazily loads each grammar and theme, then caches them. So repeated calls to codeToHtml within a single render are cheap—you don’t need to hand‑roll a highlighter cache or worry about instantiating one per block.

This caching is why the allowlist‑and‑fallback pattern works so well. Even if a page has a dozen code blocks in different languages, Shiki only loads the grammars it actually needs, and the fallback to text costs next to nothing.

The Maintenance Trap: Keeping Language Allowlists in Sync

The practical advice that saves a build: you need an explicit allowlist of supported languages, and it has to stay in sync with whatever your CMS offers as language options. Our allowlist is a 15‑entry set kept in lockstep with the language dropdown on our Payload code block. Anything not in the set falls back to text rather than throwing. Without the fallback, a single typo or an editor adding a new language in the CMS would break the entire page render.

If you’re wiring this into a rich‑text editor, there’s a related gotcha that hits you right before this one. The default markdown‑to‑Lexical converter has no transformer for a custom code block, so fenced code has to be split out and hand‑built into the right node type before conversion. That’s a separate problem from highlighting, but it’s the one you’ll hit immediately. (We addressed several Payload Lexical quirks while building this system—including a fix for GFM table rendering that you can read about in our lab post on Payload Lexical GFM tables.)

Build a Zero‑JS Blog with Us

This approach is at the core of the Next.js sites we build at techpotions. By moving highlighting to the server, we keep the client bundle lean and the Critical Path fast. If you want a performant Next.js blog that ships zero unnecessary JavaScript, explore our web development services or start a project with us.

FAQ

What happens if an unsupported language is passed to the highlighter?

The highlight function falls back to text—a built‑in Shiki language that wraps the code in <pre><code> without any syntax coloring. The page renders normally, the code is readable, and no error is thrown.

Can I use a different theme or multiple themes?

Shiki works with any theme you supply. You can switch themes by passing a different built‑in theme name or a custom JSON theme object to codeToHtml. For dark‑mode support, you can render both a light and dark theme on the server and toggle them with CSS, but that re‑introduces a tiny bit of client logic. Many teams simply pick one theme that works well in both modes.

How do I add a new language to the allowlist?

Install the corresponding language grammar (for example, npm install @shikijs/langs if you’re using the bundled approach) and add its identifier to the ALLOWED_LANGUAGES set. Then synchronise the CMS editor’s language dropdown so content authors can select it.

What the Claude API Actually Costs

techpotions — Fri, 31 Jul 2026 20:36:12 +0000

If you’re estimating the Claude API cost for a production feature, the per-model prices below give you the raw numbers—but the two structural forces that will shape your actual bill are the output-token multiplier and the expiration of introductory pricing on Sonnet 5.

(Prices verified July 2026 against Anthropic’s official page; the LLM API pricing calculator that powers these figures stamps every report with PRICING_AS_OF so the table can never quietly go stale.)

Claude API Cost: Per-Model Pricing Breakdown

Anthropic’s standard (non-batch) list prices as of July 2026. Every row was validated against platform.claude.com/docs/en/about-claude/pricing on that date.

Model	Input ($/M tokens)	Output ($/M tokens)	Notes
Claude Sonnet 5	$3 (intro: $2)	$15 (intro: $10)	Introductory pricing ends 31 August 2026
Claude Opus 5	$5	$25	—
Claude Fable 5	$10	$50	—

Two things jump out immediately: output tokens cost exactly 5× input tokens across the whole line, and Sonnet 5’s promo rate has a hard stop—modeling long-term unit economics on $2/$10 is a budget error waiting to happen.

Why Output Tokens Drive Your Claude API Bill (5× the Input Cost)

Forget the instinct to trim your system prompt; the real lever is response length. Every Claude model in this generation charges five times more per token that the API writes than per token it reads. A verbose output format will cost you far more than a long prompt ever will.

Take a typical Sonnet 5 request under intro pricing with 1 million input tokens and just half that much output:

const inputTokens  = 1_000_000;   // 1M tokens
const outputTokens =   500_000;   // 500k tokens

const inputPrice  = 2;    // $2 per million tokens (intro)
const outputPrice = 10;   // $10 per million tokens (intro)

const cost = (inputTokens  / 1e6) * inputPrice
            + (outputTokens / 1e6) * outputPrice;
// cost = $2 + $5 = $7

Even though the output is half the size, it’s responsible for over 70% of the $7 bill. When the intro rate expires and the same request hits $3/$15, the split becomes $3 + $7.50 = $10.50—still dominated by output.

If you’re building a Claude-backed feature, our AI development team spends as much time on response shaping and structured output schemas as on prompt engineering, precisely because that 5× ratio makes output length the budget’s loudest dial.

The Introductory Price Trap: Sonnet 5’s Promo Rate Expires

Anthropic’s Sonnet 5 introductory pricing—$2/M input, $10/M output—is current until 31 August 2026. On 1 September 2026, the standard rate of $3/$15 kicks in.

That’s a 50% increase on input and a 50% increase on output. Anyone building a product whose unit economics rely on the intro tier needs to bake that step-change into their runway model now. There’s no guarantee of an extension, and the standard rate is the number you’ll be paying for 99% of the product’s lifetime.

Beyond List Prices: Prompt Caching and Batch Lower Your Effective Rate

The table above shows the standard, on‑demand tier—the ceiling. In practice, two features can pull your effective Claude API cost well below the list numbers:

Prompt caching: When you re-use a large system prompt or context across many calls, caching can slash the input price for the repeated portion.
Batch processing: Submitting jobs asynchronously with longer latency windows often earns a substantial discount. The exact multiplier varies, but it’s enough to turn the list price from “budget assumption” to “worst-case scenario.”

The key: these are not footnotes. If your workload has any regularity, model your costs with caching and batch factored in, not the rack rate. The LLM API pricing calculator lets you toggle those levers on your own token volumes, so you can compare effective rates instead of guessing.

Model Your Own Usage—Don’t Trust a Worked Example

This article includes one code snippet; your own traffic patterns are the only ones that matter. Visit the LLM API pricing calculator, enter your expected monthly input and output tokens, and see a dated, per-model breakdown that you can screenshot and share with your team. If you’re ready to architect the feature on the right model from day one, start a project with us.

Note for the publisher: Every number in this article was sourced from Anthropic’s official pricing page on 2026-07-25. Before publication, re-verify all values at platform.claude.com/docs/en/about-claude/pricing and update the dateline accordingly. A pricing article with a stale table is worse than no article.

Next.js Hydration Mismatch: The SVG Gradient Trap

techpotions — Thu, 30 Jul 2026 18:49:00 +0000

If you've ever stared at a next.js hydration mismatch error with no obvious explanation, the culprit might be hiding inside an SVG gradient id. Our own studio hit this exact bug in Wordmark.tsx and BlobMark.tsx — a component that looked purely presentational, yet threw the infamous "Hydration failed because the initial UI does not match what was rendered on the server." The fix was a single import.

What a Next.js Hydration Mismatch Really Means – And Where It Can Hide

A hydration mismatch occurs when the initial client‑side render doesn’t produce the exact same HTML that the server streamed. React then can’t attach its event handlers cleanly and throws an error. The official Next.js docs explain it as "Text content does not match server‑rendered HTML," but the mismatch doesn’t have to be visible text – any attribute, key, or even an invisible id that differs will trigger it.

The classic suspects are easy to spot: rendering <time>{new Date().toISOString()}</time> or conditionally reading window.innerWidth during render. What’s far more insidious is the SVG gradient trap — a mismatch caused by a Math.random() call that lives entirely inside an attribute.

The SVG Gradient ID Trap: How Math.random() Broke Our Wordmark

We hit this in our web development studio’s Next.js project, inside a shared SVG component used for brand wordmarks and decorative blobs.

// ❌ Buggy: Math.random() produces a different id on server and client
const gradientId = `gradient-${Math.random()}`;

return (
  <svg>
    <defs>
      <linearGradient id={gradientId}>
        <stop offset="0%" stopColor="var(--color-brand)" />
        <stop offset="100%" stopColor="var(--color-accent)" stopOpacity={0} />
      </linearGradient>
    </defs>
    <text fill={`url(#${gradientId})`}>Techpotions</text>
  </svg>
);

The component looked harmless — no browser API, no locale, no date. And yet, every server render produced a different id than the client’s subsequent render. React pointed at the <svg> nesting, but the real mismatch was the id attribute itself.

The One-Line Fix: useId() Makes Ids Stable Across Server and Client

React ships useId() precisely for this scenario. It generates an identifier that is:

Stable across server and client for the same component instance.
Unique across instances, preventing duplicate ids in the DOM.

Replacing Math.random() with useId() eliminated the hydration mismatch entirely.

// ✅ Fixed: useId() from React is stable across server and client
import { useId } from 'react';

const gradientId = useId();

return (
  <svg>
    <defs>
      <linearGradient id={gradientId}>
        {/* ... stops ... */}
      </linearGradient>
    </defs>
    <text fill={`url(#${gradientId})`}>Techpotions</text>
  </svg>
);

No suppressHydrationWarning, no useEffect gymnastics. Just one hook. We’ve since made this a standing rule in our project instructions: no Math.random() in an SVG linearGradient id, use useId().

The General Rule: Any Impure Render Value Is a Hydration Time Bomb

This bug teaches a larger lesson worth internalising: a hydration mismatch is any render-time value that is not a pure function of props and state. Critically, these values can hide in attributes you wouldn’t think to audit:

Impure Source	Example	Why It Triggers a Mismatch
`Math.random()`	`id={`item-${Math.random()}`}`	Server and client get different random numbers
`Date.now()` / `new Date()`	`<time>{Date.now()}</time>`	Timestamp differs between renders
`localStorage`	`<div>{localStorage.getItem('theme')}</div>`	Only available on the client
`window.innerWidth`	`<Layout width={window.innerWidth}>`	`window` is undefined on the server
`navigator.language`	`<Document lang={navigator.language}>`	Server has no navigator API
Request‑time data	`fetch('/api/user', req)`	User‑specific data without proper serialisation

As the community on Stack Overflow affirms, even random ids inside SVGs can cause the “initial UI does not match” error. The error message pointing at surrounding markup is often a red herring — the real source lives in an attribute value you’d almost never inspect.

Debugging Strategy: Grep for Math.random, Date, and Window Before Bisecting

When the error names a component that looks pure, don’t start bisecting your component tree yet. Instead, run:

grep -r 'Math.random' --include="*.tsx" src/components/marks/
grep -r 'Date(' --include="*.tsx" src/
grep -r 'window.' --include="*.tsx" src/

In our case, the offending call was a single word inside a template literal in an id attribute — easy to miss in manual review, but instantly caught by a simple grep. If you’re in a monorepo or using complex third‑party libraries, this search can save hours of frustration.

How We Prevent Hydration Mismatches at techpotions

After shipping the useId() fix, we added a lint rule that flags any Math.random() inside JSX‑returning functions. It’s one of several defensive practices we bake into our Next.js projects. For example, when we built a smooth‑scroll animation that needed to work on Windows, we fixed a Lenis scroll issue by ensuring the client‑only logic never leaked into the server render.

If your app is plagued by hydration errors you can’t trace, our web development team offers code‑reviews and focused debugging sessions. We focus on the attribute‑level detail that generic tooling often misses.

Ready to start? Let’s make your Next.js app bulletproof.

FAQ

Why does Math.random() cause a hydration mismatch?

Because server and client rendering are independent processes. Math.random() returns a different number each time, so the generated gradient id is different on the server than on the client. React then detects that the initial HTML doesn’t match what the client expects and throws the mismatch error.

How does useId() solve the problem?

React’s useId() generates a stable, deterministic identifier that is guaranteed to be identical on the server and client for the same component instance. It doesn’t rely on impure data, so the id matches every time.

Can other attributes aside from id trigger a hydration mismatch?

Absolutely. Any attribute — class names, keys, data-* attributes, inline styles, or even render‑time logic that affects markup structure — can trigger a mismatch if it derives from impure sources like Date.now(), localStorage, window, or request‑time data. Treat every attribute as a potential leak point.

Getting Reliable JSON Out of an LLM in Production

techpotions — Thu, 30 Jul 2026 18:48:33 +0000

LLM structured output JSON schema is the foundation of any production pipeline that relies on machine‑readable responses. But even with a strict schema and a well‑tuned prompt, our daily content‑generation pipeline broke twice—once because a provider silently dropped support, and once because a widely‑recommended parsing trick ate our own data. Neither failure appeared in local testing. Both teach the same lesson: a structured output contract is not a reliability guarantee; you must still code for the real world.

Why LLM JSON Parsing Still Breaks in Production

The promise of llm structured output json schema is that you get a pristine JSON object that matches your definition. In practice, two things go wrong. First, not every model gateway supports json_schema with strict: true; some return HTTP 400. Second, many parsers blindly strip markdown code fences, a heuristic that corrupts any field that legitimately contains triple‑backtick code blocks—like the Markdown body of a generated article. We hit both in a pipeline that runs daily behind our AI‑powered comparison pages.

The Right Way to Request Structured Output

Start by setting response_format to { type: "json_schema", json_schema: { strict: true, schema: yourSchema } }. This eliminates prose drift and guarantees valid JSON shape. But assume nothing: read the model name at call time so a runtime environment override can swap models without a redeploy. If you hardcode the model at module load, your only escape during an outage is a full deploy.

// Always resolve the model at call time, not at module load
export async function getStructuredCompletion(
  messages: ChatMessage[],
  schema: Record<string, unknown>,
  model = process.env.STRUCTURED_MODEL || "gpt-4o",
): Promise<Record<string, unknown>> {
  const body = {
    model, // <-- runtime override friendly
    messages,
    response_format: {
      type: "json_schema",
      json_schema: {
        strict: true,
        schema,
      },
    },
  };
  // …
}

Failure Mode #1: Provider Capability Drift

Even when your API client sends a valid json_schema request, an upstream provider may reject it. Some gateways don’t support strict: true at all and return HTTP 400. Our client catches that exact status and retries with response_format: { type: "json_object" }. The model then outputs unconstrained JSON (possibly wrapped in prose), so we follow up with a tolerant parser. The key insight: strict json_schema is the right default, not a lowest‑common‑denominator compromise. Detect and degrade, never pre‑weaken your request.

try {
  const res = await fetch(this.baseUrl, {
    method: "POST",
    headers: this.headers,
    body: JSON.stringify({
      ...body,
      response_format: {
        type: "json_schema",
        json_schema: { strict: true, schema },
      },
    }),
    signal,
  });
  if (res.status === 400) {
    // Provider rejected strict schema; fall back to json_object
    console.warn("400 on strict json_schema; retrying with json_object");
    const fallbackRes = await fetch(this.baseUrl, {
      method: "POST",
      headers: this.headers,
      body: JSON.stringify({
        ...body,
        response_format: { type: "json_object" },
      }),
      signal,
    });
    return this.handleJsonResponse(fallbackRes, schema);
  }
  return this.handleJsonResponse(res, schema);
} catch (e) {
  // …
}

Failure Mode #2: When Fence‑Stripping Parsers Mangle Content

The usual advice for parsing LLM JSON is to strip triple‑backtick fences before calling JSON.parse. That heuristic is actively dangerous when any field in your schema contains fenced code blocks. Our blog generation pipeline produces articles whose markdown field routinely includes code blocks. A fence‑stripper that greedily removes everything between backticks will corrupt the payload. We learned this the hard way when a generated article lost half its code examples.

Our parser does the opposite: it tries JSON.parse(raw) first, because native structured output should never produce fences. Only if that initial parse throws do we extract the substring from the first { to the last }. We never remove backticks.

function tolerantParseJson(raw: string): Record<string, unknown> {
  // Try direct parse first (works for native structured output)
  try { return JSON.parse(raw); } catch { /* fall through */ }

  // Fallback: grab everything between the first { and the last }
  const firstBrace = raw.indexOf("{");
  const lastBrace = raw.lastIndexOf("}");
  if (firstBrace !== -1 && lastBrace !== -1 && lastBrace > firstBrace) {
    const extracted = raw.slice(firstBrace, lastBrace + 1);
    return JSON.parse(extracted);
  }

  throw new Error("No JSON object found");
}

This approach preserves embedded code fences, handles models that occasionally wrap JSON in a single pair of backticks (the braces still exist), and doesn’t penalize providers that return pristine JSON.

Additional Patterns That Keep Pipelines Running

Keep the system prompt explicit about forbidden invention. Structured output guarantees shape, not truth. Our prompt includes: “If a requested fact is not supported by the provided sources, leave it out rather than inventing it.” Schema compliance and factual accuracy are unrelated problems; mixing them leads to beautifully‑typed hallucinations.

Add a lightweight evaluation step for mission‑critical pipelines. Even after parsing succeeds, run a few assertions against the output—expected keys, value types, content‑based rules. This catches the infamous “valid JSON, wrong content” failure without a heavy ML eval framework.

Log every raw response (sanitized if needed) for at least a few days. When parsing fails in production, the first question is always “what did the model actually return?” Having the raw payload in your logs turns a mystery into a five‑minute fix.

FAQ

Why does my LLM return 400 when I request json_schema with strict mode?

Some providers or gateways don’t support strict: true. The fix is to catch the 400 status and retry the same messages with response_format: { type: "json_object" }, then use a tolerant parser that handles unconstrained output.

Can’t I just strip triple backticks before parsing JSON from an LLM?

Avoid blanket fence‑stripping. If any field in your schema can contain backtick code blocks (like Markdown), the stripping logic will corrupt the payload. Instead, attempt JSON.parse directly, then fall back to extracting text between the first { and last } if that fails.

Does structured output prevent hallucinations?

No. Structured output guarantees that the JSON matches your schema, but not that the content is correct. Always prompt the model to omit unsupported facts rather than invent them, and consider adding a post‑parse validation or eval step for high‑stakes pipelines.

Block AI Crawlers: The 15 Bots That Matter

techpotions — Wed, 29 Jul 2026 18:34:25 +0000

Most lists that claim to help you block AI crawlers are copy-pasted and dangerously wrong about the two tokens that actually matter. Sorting them properly is not an abstract taxonomy exercise. It is the single decision that determines whether your content vanishes from AI answers while training continues — or vice versa.

We maintain the crawler registry in lib/ai-crawlers.ts that powers techpotions’ free AI robots.txt generator. Every agent string and description was verified against the operator’s own crawler documentation. The registry holds 15 verified bots across four categories, and that four-way split is this article’s structure, because the categories map directly to what blocking costs you.

Two tokens almost everyone gets wrong

Before the list, the single most important correction to make, and almost every listicle on this query gets it wrong: Google-Extended and Applebot-Extended are not crawlers. They are robots.txt tokens — product controls that govern whether your content is used for Gemini and Apple foundation-model training.

Blocking Google-Extended does not affect Google Search crawling, Google ranking, or regular Applebot search indexing. People block them believing they are opting out of AI Overviews, and are actually opting out of nothing they think they are, while leaving search indexing completely untouched.

Platforms have started wiring these tokens into one-click controls. Cloudflare’s managed robots feature, released mid-2025, lets you add AI crawler rules through a dashboard toggle rather than editing a raw file — but the underlying token logic above still applies.

Block AI crawlers: the four categories that decide the cost

Every AI crawler we track belongs to one of four categories. The category tells you the cost of blocking it.

Training crawlers scrape pages to feed a model that may never cite you. Blocking them is a defensive data decision.
Assistant crawlers fetch pages to answer a live user’s question and can cite and link you. Blocking them removes you from AI answers.
Search crawlers underpin traditional or AI search indexes. Blocking them is a visibility trade-off with ranking consequences.
Scrapers operate with little transparency and rarely honor robots.txt. A block is a declaration of intent, not a guarantee of enforcement.

Blocking everything is the lazy answer, and it costs AI-answer visibility. The entire point of this list is to let you decide bot by bot, with your eyes open.

Training crawlers: default-deny if you opt out of training

These bots feed foundation models. They will not send you traffic or citations. The trade-off is simple: if you want to block AI crawlers from training on your work, start here.

Crawler	Operator	What it does	What you give up by blocking it
GPTBot	OpenAI	Crawls content that may be used to train OpenAI’s generative foundation models, including the models powering ChatGPT.	The clearest training opt-out. Blocking this has zero impact on whether ChatGPT can answer a live user query about your page — that is a separate assistant crawler.
CCBot	Common Crawl	Crawls the web to build the Common Crawl dataset, a massive open repository of web pages used as training data by many LLM builders.	You opt out of a foundational training dataset used across the industry, not just a single model. It sends no user traffic.
ClaudeBot	Anthropic	Crawls content that may be used to train Claude. Separate from Anthropic’s assistant crawler.	You lose no AI-answer visibility. Claude’s live-fetch assistant crawler is a completely different agent.
Google-Extended	Google	A token, not a bot. Controls whether content crawled by Google’s regular search crawler can be used for Gemini foundation-model training.	Opting out does not affect ranking, Google Search crawling, indexing, or inclusion in AI Overviews. You are only blocking Gemini training.
Applebot-Extended	Apple	A token, not a bot. Controls whether content crawled by regular Applebot can be used for Apple foundation-model training.	Does not affect Apple search indexing. Blocking this does nothing to Siri or Apple Intelligence live answers.

Assistant crawlers: block these and you leave the conversation

Assistant crawlers fetch your page to answer a real user’s question. They can cite and link you. This is the category where blanket blocking most often hurts traffic, because the site owner thought they were stopping training.

Crawler	Operator	What it does	What you give up by blocking it
ChatGPT-User	OpenAI	Fetches pages on behalf of ChatGPT users who ask questions that require live web retrieval. It cites sources.	You remove your content from ChatGPT’s answers entirely. No citation, no referral traffic, no visibility in the assistant that millions use for real-time research.
OAI-SearchBot	OpenAI	Crawls for OpenAI’s web search product. Used for surface-level search indexing rather than training.	You disappear from OpenAI’s native search experience, distinct from ChatGPT’s browsing. An SEO visibility call, not a training call.
PerplexityBot	Perplexity	Fetches pages to generate Perplexity’s AI answers with inline citations.	Blocking it means Perplexity cannot pull your page into answers, removing a growing source of cited traffic.
Cohere-AI	Cohere	Fetches web content to power Cohere’s Retrieval-Augmented Generation pipeline, with citations.	You drop out of an enterprise RAG stack. If your content answers high-stakes or business queries, this is a concrete loss.

Search crawlers: the visibility trade-off

These bots feed search indexes, including AI search. Blocking them carries ranking consequences — not theoretical ones.

Crawler	Operator	What it does	What you give up by blocking it
Googlebot	Google	Decades-old search crawler powering Google Search, Discover, and now the AI Overviews index.	You leave the Google search ecosystem entirely. There is no separate token to opt out of AI Overviews while keeping search ranking. Googlebot feeds both.
Bingbot	Microsoft	Crawls for Bing Search and powers the retrieval layer behind Microsoft Copilot answers.	You vanish from Bing Search and from Copilot citations. One block, two surfaces.
DuckDuckBot	DuckDuckGo	Crawls for DuckDuckGo’s web index, which also underpins DDG’s AI answer features.	You leave a search engine that, while smaller, fuels AI answers that do cite. Visibility trade-off.

Scrapers: a block is the right signal, but enforcement is uneven

These crawlers aggregate content, often for commercial datasets or internal models. The compliance story is thinner, and four entries in our registry carry explicit caveats. Bytespider is the clearest case: it has no official crawler documentation at all, so our entry states exactly that rather than pretending a policy exists. A block here signals intent. It is not a technical guarantee.

Crawler	Operator	What it does	What you give up by blocking it
Bytespider	ByteDance	Widely observed to crawl for TikTok and Douyin recommendations, potentially feeding internal research datasets. No official docs exist.	There is no documented user-facing benefit to keeping it. The cost is speculative scraping with zero transparency.
FacebookBot	Meta	Crawls for article previews, ad quality, and internal ML research. The boundary is deliberately blurred.	You lose rich Facebook share cards and article previews. The same agent may also feed training, and Meta does not separate them with distinct tokens.
Amazonbot	Amazon	Crawls for Alexa, product search, and AWS AI services. Amazon’s documentation states metadata enrichment as a secondary purpose.	Blocking it can affect how your products appear in Alexa answers and the Amazon shopping index, but the crawl does not offer referral traffic back to your site.

How to block AI crawlers without breaking visibility

The right approach is a per-bot robots.txt rule set that maps to your actual goals. A site that wants training opt-out but AI-answer visibility might look like this:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

This blocks foundation-model training while keeping your content in ChatGPT and Perplexity answers. We built the AI robots.txt generator to produce these split-category files in seconds, because the mistake we kept seeing was people pasting a blockade that also removed them from AI answers. For the specification behind agent instructions beyond robots.txt, see llms.txt examples.

If you are tuning this as part of a broader AI visibility play, our Growth Labs team works on the question from the traffic side: what you gain by being in the models people are actually asking.

FAQ

Does blocking Google-Extended stop my content appearing in AI Overviews?

No. Blocking Google-Extended tells Google not to use crawled content for Gemini foundation-model training. It does not affect Google Search crawling, ranking, or inclusion in AI Overviews. Blocking it does not opt you out of AI Overviews.

Can blocking AI crawlers hurt my visibility in AI-powered answers?

Yes. Assistant crawlers like ChatGPT-User fetch pages to answer a live user’s question and can cite and link you. If you block the assistant category, those tools cannot retrieve your content at all, and you become invisible in their answers. The typical mistake is a blanket block that removes sites from AI answers while the owner intended only to opt out of training.

Why can’t I just use a NoAI meta tag to stop training?

Because it is not a reliable mechanism against the scrapers that ignore it. Bytespider has no official crawler documentation, so there is no verifiable opt-out procedure. A <meta name="robots" content="noai"> tag is not honored by all training crawlers and offers no protection against bad-faith scraping. A robots.txt rule is the stronger, more widely recognized signal.

Ready to set the exact rules that match your risk tolerance and visibility goals? Start with our crawler registry tool.

Next.js Sitemap Not Updating? Here's the Real Fix

techpotions — Wed, 29 Jul 2026 18:34:14 +0000

Next.js Sitemap Not Updating? Here's the Real Fix

If your Next.js sitemap is not updating after you publish new content, you're dealing with a cache-coherence bug that almost nobody writes up. It has an exact symptom, a reproducible root cause, and a one-line fix. This is the guide you'll wish you had the moment you notice /sitemap.xml serving fewer entries than your real site.

The symptom: your sitemap lags behind your published content

The mismatch is impossible to miss once you look. On our own site, /lab lists 11 published posts, yet /sitemap.xml shows only 7. Same database, same deploy, two different answers. If that gap sounds familiar, you're in the right place.

You might have checked your afterChange hook, verified that revalidateTag('posts') fires, and even confirmed that the tagged data refreshes — only to find the sitemap still frozen. That's because the problem lives between two cache layers, not inside the data fetch.

Why revalidateTag doesn't fix a stale Next.js sitemap

The answer lies in what sitemap.ts actually is. According to the Next.js Metadata Files: sitemap.xml documentation, it's a special Route Handler. And like any Route Handler, Next.js caches its rendered output by default.

Here's what happened in our own repository (this bug is documented in a comment at the top of app/(frontend)/sitemap.ts because it cost real indexation time):

We read content from Payload using unstable_cache, tagged with the collection slug posts.
An afterChange hook called revalidateTag('posts') whenever a post was published.
That call did work — it invalidated the inner unstable_cache data entry.
But the route's statically-rendered outer XML output was never re-run. The frozen route output kept serving the old XML built from the old data, long after the inner cache was refreshed.

Two cache layers. Tag-based revalidation busted the inner one, but the outer route handler cache was never told to re-execute. That's the missing piece.

The one-line fix: route-level ISR on the sitemap

Tell the sitemap route to re-render on a bounded interval:

// app/(frontend)/sitemap.ts

export const revalidate = 3600; // seconds — re-render at most once per hour

That's it. With this line, the whole route re-executes after 3600 seconds and always picks up newly published documents. The sitemap is crawler-only traffic, so an hour of staleness is irrelevant. Even better, the bounded interval deduplicates the Payload read to at most once per hour, so you're not hammering the database on every bot request.

Once route-level ISR is in place, the inner unstable_cache becomes redundant. The route itself is now invalidating on a schedule, so you can drop the wrapper and simplify your data fetching.

Drop the inner cache after adding revalidate

Before, your sitemap might have looked like this:

import { unstable_cache } from 'next/cache';
import { getPayloadPosts } from '@/lib/payload';

export async function GET() {
  const posts = await unstable_cache(
    getPayloadPosts,
    ['sitemap-posts'],
    { tags: ['posts'] }
  )();

  // generate XML…
}

After setting export const revalidate = 3600, you can remove the unstable_cache wrapper entirely:

export const revalidate = 3600;

export async function GET() {
  const posts = await getPayloadPosts();
  // generate XML…
}

The route-level ISR already ensures the handler re-runs periodically, and on-demand revalidation can still be triggered with revalidatePath('/sitemap.xml') from your publish hook if you need an instant refresh. No more stale XML.

The general lesson: tag-based revalidation invalidates data, not rendered output

This bug taught us a rule that applies to every part of Next.js where you stack a data cache inside a route cache:

revalidateTag clears your data, not the route that renders it. If the route itself is cached (static generation, full route cache, etc.), invalidating inner data leaves the outer rendered output untouched.

Whenever you have both layers, you must either:

set route-level revalidate to give the outer layer a time bound, or
call revalidatePath on the route itself to manually bust the rendered output.

This is the piece that the revalidateTag documentation doesn't spell out, and it's the difference between a sitemap that stays fresh and one that silently sabotages your indexation.

After fixing your sitemap, tell search engines about the new content immediately — our IndexNow integration for Next.js pushes updates the moment you publish.

Struggling with caching or Next.js performance in general? Our team builds products where this stuff is ironed out before it bites you — start a project and we'll make sure your site's infrastructure stays boringly reliable.

FAQ

Why is my Next.js sitemap not updating even though revalidateTag works?

revalidateTag only clears data caches tagged by unstable_cache. Your sitemap route's rendered output is cached separately as a static route handler. To update the sitemap, you need to either revalidate the path with revalidatePath('/sitemap.xml') or set route-level ISR with export const revalidate.

Should I use unstable_cache in my sitemap route after adding revalidate?

No. After setting revalidate on the route, Next.js will re-render the whole handler periodically, making the inner data cache redundant. Remove unstable_cache to simplify.

What's the recommended revalidate interval for a sitemap?

An hour (3600 seconds) is safe. Sitemaps are consumed by crawlers sporadically, so a small delay won't hurt indexation, while it protects your database from unnecessary reads.

What AI-Assisted Posts Actually Do for a New Domain

techpotions — Tue, 28 Jul 2026 18:28:50 +0000

Do AI generated blog posts rank? We ran our own AI-assisted publishing pipeline on a brand-new domain (techpotions.com) for six months, and the honest answer is: not yet, and not for the reasons you think. The raw numbers are unflattering, but the diagnosis—and the rebuild that followed—is what matters for any founder being sold an AI content engine.

What our zero‑traffic AI content pipeline actually looked like

Our baseline, pulled in July 2026, was brutal. Semrush showed zero organic traffic, an Authority Score of 2, and exactly one genuine referring domain (a nofollowed Medium mention). Google Search Console revealed a tiny trickle over 28 days: 16 clicks from 341 impressions at an average position of 12.1. We had published roughly 40 posts to our /lab section, but the numbers told a different story behind the scenes.

Metric	Value
Semrush Authority Score	2
Organic traffic (Semrush)	0
Organic keywords (Semrush)	4
Referring domains (genuine)	1
Pages indexed (GSC)	47
Pages not indexed (GSC)	56
Clicks (28 days, GSC)	16
Impressions (28 days, GSC)	341
Avg. position (GSC)	12.1

Those 56 unindexed URLs weren’t a sign of low-quality content being rejected. They were a sign of a broken technical foundation. 33 of them were www duplicates of pages that already existed at the apex domain—so they were never separate content at all. Another 2 were stale redirects left over from a hosting migration. The rest were thin or orphaned pages that the generator had produced in volume mode.

Volume was never the constraint. The pipeline was a volume machine, recycling trending news into posts that summarized what was already ranked elsewhere. It was writing, but it wasn’t saying anything that hadn’t been said a hundred times.

Do AI generated blog posts rank? Here’s why ours didn’t.

Three things were broken, and none of them were fixed by writing more.

1. Indexation sabotage. Before you measure traffic, measure indexation. Of our 56 unindexed URLs, 35 were completely avoidable—33 www duplicates and 2 stale redirects. Google had crawled the right pages but chose to ignore the wrong ones, and we had given it twice as many wrong ones as right ones. When you fix that, you don’t get better content; you just stop actively sabotaging your own presence. We cleaned up canonical tags, set up proper redirects, and used IndexNow for Next.js to tell search engines about the canonical versions immediately.

2. Domain authority gating. Authority Score 2 with one real referring domain does not rank, regardless of how good any individual post is. No amount of publishing fixes a referring-domain problem. This is the gap that most AI content engines conveniently ignore. SE Ranking’s own experiment with AI-generated articles on an established blog saw traffic gains precisely because the domain already had authority. Without that, even well-written, helpful AI-assisted posts remain invisible.

3. Undifferentiated output. Our earlier pipeline recycled trending news. It produced posts that merely summarized what was already ranking. That’s the fastest way to hit the “thin content” wall, even if Google’s guidance on AI content says it’s not automatically spam. The posts weren’t deceptive; they were just redundant. As one Reddit thread points out, the best AI-assisted workflows use research and first-hand data, not just summarization. We had to kill the news-recycling mode entirely.

The pipeline rebuild that stopped writing when there was nothing to say

We rebuilt the generator so that it will only draft a topic that carries a real proof point and a first-hand expert note. If the backlog has neither, the generator produces zero posts by design. An empty backlog now means zero output.

# Pseudo-code for the rule we baked into the pipeline
def should_generate(topic):
    if not (topic.get("proof_point") and topic.get("expert_note")):
        return None  # No output until a real expert note exists
    return draft_post(topic)

This is the difference between “using AI to write” and “using AI to scale what you already know.” Averi’s breakdown gets it right: AI-generated content isn’t good or bad for SEO; your process for creating AI-assisted content determines whether it helps or hurts. Our process now requires a human to bring something original before the model gets to type a single word. We also built in safeguards against the hallucination problem that plagues AI pipelines—stopping hallucinations before they publish is a non-negotiable layer.

Transferable rules for founders buying an AI content engine

Measure indexation before you measure traffic. Unindexed posts are not slow—they are absent. If half your pages aren’t in Google’s index, no amount of content tweaking will help. Fix duplicate-host and redirect issues before blaming the copy.

Judge month one on indexation and keyword coverage, not sessions. A new domain with zero authority will not earn meaningful traffic in the first month, no matter how good the content is. Track how many pages are indexed and how many queries they show up for in Search Console—that’s your real leading indicator.

AI removes the drafting cost, not the authority cost. The CreativeStable experiment with automated AI blogging hit the same wall: volume alone doesn’t build trust. You still need to earn links the old-fashioned way. AI-assisted writing changes your publishing rate and nothing about your link profile.

Require first-hand proof points or the pipeline stays silent. If your topic backlog is just a list of keywords, you’ll produce the same summary everyone else has. The only AI content that earns a place on the web is built on something the model didn’t read on the internet—your data, your experiment, your expert note.

These rules are exactly what we’ve baked into Growth Labs, where we pair AI-assisted drafting with real authority-building tactics so founders don’t have to learn the hard way.

How to start building an AI-assisted content engine that actually works

Clean up your technical foundation. Fix canonicalization, eliminate www duplicates, and remove stale redirects. Use IndexNow to push canonical URLs to search engines the moment you publish.
Build a topic backlog that requires proof points. Every topic should have a “why us” column: a data point, a screenshot, a customer story, a build log. If you can’t fill that column, the topic doesn’t get written.
Set the generator to produce zero output when the backlog is empty. The pipeline should not have a fallback mode that summarizes trending news. “No expert note” means “no post.”
Track indexation and keyword coverage, not sessions. In month one, your goal is to get 90%+ of your pages indexed and to see impressions climbing for a growing set of queries. Traffic comes later.
Invest in link building separately. The AI pipeline is your drafting engine; you still need a link-building strategy. Treat them as two independent workstreams.

Ready to build an AI-assisted content engine without the expensive mistakes? Start here.

FAQ

Do AI generated blog posts rank on a new domain?

Not automatically. On a brand-new domain with zero authority, even genuinely helpful AI-assisted posts won’t rank if they are not indexed, if the technical foundation is broken, or if the content itself is undifferentiated. Our own pipeline produced zero organic traffic according to Semrush, despite 40 posts, because indexation issues and domain authority gate kept them invisible.

What’s the first thing to fix before using AI-generated content on my site?

Fix indexation first. Before writing a single post, clean up duplicate canonical URLs, stale redirects, and www/non-www conflicts. Then build a topic backlog that requires a first-hand proof point and an expert note, and set your generator to produce zero posts when the backlog is empty. Finally, track keyword coverage and indexed pages, not sessions, in month one.

Can I rank a new domain with AI content alone, without any link building?

No. AI removes the drafting cost, not the authority cost. A domain at Authority Score 2 with one real referring domain will not rank, regardless of post quality. You need a separate link-building strategy alongside AI-assisted publishing to earn the referring domains that signal trust.

DEV Community: techpotions

SearXNG JSON API for LLM Grounding

SearXNG JSON API Endpoint

Parameters that earn their keep

The raw TypeScript call

What It Replaced: Paid Search Markup

The Resilience Pattern: Degrade, Don't Fail

Two Jobs the SearXNG JSON API Handles

1. SERP-aware framing

2. Source grounding

The Honest Caveat

Getting SearXNG Running

FAQ

What is the SearXNG JSON API?

Is the SearXNG JSON API free?

Why use SearXNG instead of a paid search API for AI agents?

Can I use the SearXNG JSON API in production?

When to Turn Off Next.js Image Optimization

When next.js images unoptimized is the correct call

When you must keep Next.js image optimization enabled

How to configure next.js images unoptimized (and why it needs a comment)

AI Automation Agencies: Real vs Repackaged

How to separate genuine AI automation agencies from resellers

Ask where the human approval gate lives

The real tell: what an AI automation agency refuses to automate

Syndication doesn’t buy you backlinks — and honest agencies say so

What to look for in your next conversation

FAQ

How can I tell if an AI automation agency actually runs its own workflows?

Where should human review sit in an automation pipeline?

Do syndicated blog posts help with SEO backlinks?

What Agentic AI Development Services Actually Ship

What Agentic AI Development Services Actually Deliver

1. An Eval Suite That Ships Before the Agent

2. A Structured Output Contract, Not Free Text

3. A Defined Human Handoff Point

4. Grounding Infrastructure That Never Takes Generation Down

5. Observability on the Ops Surface

Why the Model Call Is the Small Part

FAQ

What do agentic AI development services actually cost?

How long does it take to build an agentic AI product?

Do I need structured output for my agent?

When Not to Use an AI Agent

When Not to Use an AI Agent

Case 1: Publishing Fan‑Out — When Every Step Is Known, Write a Pipeline, Not an Agent

Case 2: Content Ideation — Constrain the System, Don’t Optimize the Agent

Case 3: Candidate Screening — Automate the Reading, Not the Judgment

The Decision Rule: Agent, Deterministic Code, or Human

When Selling Agents Means Telling Clients “Don’t Build One”

FAQ

What is the simplest test for when not to use an AI agent?

How is an AI agent different from a chatbot, and why does that matter?

Can I use n8n instead of building a full AI agent?

Shiki in Next.js App Router: Zero Client JS

Shiki + Next.js App Router: Zero KB of Client JS

The Core Move: Highlighting in a Server Component

Why This Approach Scales: Shiki’s Internal Caching

The Maintenance Trap: Keeping Language Allowlists in Sync

Build a Zero‑JS Blog with Us

FAQ

What happens if an unsupported language is passed to the highlighter?

Can I use a different theme or multiple themes?

How do I add a new language to the allowlist?

What the Claude API Actually Costs

Claude API Cost: Per-Model Pricing Breakdown

Why Output Tokens Drive Your Claude API Bill (5× the Input Cost)

The Introductory Price Trap: Sonnet 5’s Promo Rate Expires

Beyond List Prices: Prompt Caching and Batch Lower Your Effective Rate

Model Your Own Usage—Don’t Trust a Worked Example

Next.js Hydration Mismatch: The SVG Gradient Trap

What a Next.js Hydration Mismatch Really Means – And Where It Can Hide

The SVG Gradient ID Trap: How Math.random() Broke Our Wordmark

The One-Line Fix: useId() Makes Ids Stable Across Server and Client

The General Rule: Any Impure Render Value Is a Hydration Time Bomb

Debugging Strategy: Grep for Math.random, Date, and Window Before Bisecting

How We Prevent Hydration Mismatches at techpotions

FAQ

Why does Math.random() cause a hydration mismatch?

How does useId() solve the problem?

When `next.js images unoptimized` is the correct call

How to configure `next.js images unoptimized` (and why it needs a comment)